Joe Gray - Dear Blue Team: Forensics Advice to Supercharge your DFIR Capabilities and Timing

all right cool so welcome to dear blue team so basically I'm just trying to provide a little bit of advice from a forensics point of view for non forensics people to basically expedite your capabilities before we get started the thoughts and opinions that I express today do not really reflect those of IBM I say this because I'm a senior security architect at IBM I'm also 2017 derbycon social engineering capture the flag winner member of the no lacan third place ascent capture the flag team I write for Forbes I used to navigate submarines I'm writing a book with no starch that's about it with that so basically why this topic in this talk so I was working on sans forensics 508

which corresponds to G CFA basically it's advanced Incident Response digital forensics and threat hunting and basically I'm somewhat of an academic at heart so I always try to synthesize the synergy and cohesion okay I was just playing buzzword bingo there for a second but I do try to synthesize the things that I that I'm learning and find application for it sometimes in a use case scenario sometimes in the abuse case scenario it's just whichever is more appropriate in this case it's a use case scenario so basically this is the whole outcome of those thoughts so to start out with like baseline knowledge just to level the playing field d fear DF ir it's digital forensics an incident

response basically Incident Response is the big picture of what you're doing to overcome an adverse event so an incident is an adverse event of an event is anything that happens on an information system incident handling is also the big picture even even more so than incident response because Incident Response is considered the technical components whereas incident handling brings in the business unit may bring in legal and some other organizations and that's per the Internet storm center so we commonly hear about pike curl that's the actual incident response process NIST has a similar process but it's not quite as catchy in terms of nomenclature as pike girl so basically it's preparation identification containment eradication recovery lessons

learned so basically at any given time if you're not actively involved in an incident you are in the preparation phase or maybe on the cusp of lessons learned in preparation depending on when the last incident occurred so something that we also put into the equation is threat intelligence so where does it fit in is threat intelligence derivative of incident response or is it consumed by Incident Response so I mean with that the answer to that is yes so especially if you're doing like threat hunting you're certainly going to be consuming threat intelligence but if you're responding to something you may be producing it or comparing it against someone else's threat intelligence to try to do that pesky attribution that

every vendor wants to wants everyone to believe that they're doing so with regards to that yet I would say that D fear and threat intelligence in this case I used CT I for cyber threat intelligence are they one of the same no there they're complementary to each other so a preconceived notion that I had taking the course I had didn't really have a whole lot of forensics knowledge going in and I was familiar with like file carving and things like file system forensics so anything you could do with like sleuth kit autopsy that kind of stuff I didn't even think about network and memory forensics so a lot of this is actually heavy on the

memory forensics side so I've detailed a few tools that you could play with actually they're free as well the only one that could cost would be network miner if you pay for the license with that but the free version is a pretty hardcore in itself but anyway if you download sift the sands investigative forensic toolkit most of the things are already there there are other distributions similar to sift one used to be kein the computer-aided investigative networking environment it's not been maintained so I mean sift is pretty much the most up-to-date one that you're going to get most of these things are already there except for the things that have to be Windows like network miner so

and to some degree Kali has the capability I mean I don't necessarily think I would use Kali for that but if you're in a pinch and that's all you've got it'll work you've got volatility there you've got some of the other tools but they're not necessarily there for the purpose of doing forensics they're more there for more nefarious reasons also we have rim necks the reverse engineering malware distribution you can install it on top of sift or sift on top of it whichever you see fit so you could play the two off of each other but anyway the third type of forensics aside from file system and memory would be networking that's dealing with breaking

pcaps up figuring out what's going on there and the one thing to think about with this is I know by the talk title and I deliberately did this because I'm a huge fan of clickbait talk titles this is not a thread hunting presentation so the proactive advice I'm giving you has nothing to do with like thread hunting I'll discuss it on this slide in the next but it's not something that we're gonna go into the fine details with so I mean when we talk about thread hunting and we want to look at tools and techniques a lot of a lot of the tools and techniques you're going to use for thread hunting are identical to those

that you're going to use in responding to an incident basically thread hunting versus forensics thread hunting is before you know you have an incident forensics is after you know you have one so it's pretty much the same thing there's a lot of benefits to it because it's going to expedite your incident response time it's going to help you maintain some cleaner architecture probably save some brand reputation damage that could come about but the problem with that is maturity using the Center for internet securities critical security controls formerly known as the sans top 20 if you're not meeting about 15 of those you're probably not at a maturity level to even invest in that because thread hunting is based about on

what is deviating from what is considered normal if you don't have say an inventory of authorized and unauthorized software or authorized an unauthorized Hardware how can you do hunt operations against it if you don't know what's there if you don't have your access management in check and everybody has a local admin what's the point you've got other fish to fry so you need that level of maturity with it so looking at the comparison for forensics it's very reactive you don't really need the prerequisites like you would with threat hunting sometimes you're notified via law enforcement could be monitoring it could be activity I'll specifically phrase this a certain way Jason Street used to say that a certain

journalist should not be your IDs so in light of the whole doxxing thing that happened I'm just gonna say that journalists should not be your IDs but basically the aim of forensics like the D fair process is to get things back up and running as quickly as possible because time is money and most businesses want to get everything back up out of the incident scenario and back to normal as quickly as possible and that creates a huge dilemma between the security office and the rest of the business not that there was already any tension between security and the business but it's there so again with threat hunting you've got your maturity your size your capabilities if you if

you don't have that maturity level then you probably don't need to do it if you don't have your own Incident Response Team it may be better that you don't do it because you can hire a consulting firm to come in and give them 40 60 80 hours to do it but you've got to factor in that they aren't they're only gonna know what you tell them and that could be derivative of what you volunteer them or what they ask they're probably not going to have the intricate knowledge of your architecture to understand normality so as a by-product is that you're going to kind of sell yourself a little bit short if you need to consult

someone to do it you could but if you have your own incident response team that understands what is normal and maybe even throw a few system administrators maybe a network administrator on that team you're gonna get far better results again it's proactive in nature it is based around what's normal and a lot of times it's very similar to a red teaming gage Minh a red team engagement versus a pen test is focusing on a specific objective such as like detection times detection capabilities it's not just trying to get in get domain admin before lunch and ring the bell right so with this you're taking a specific piece of threat intelligence and you are searching for it you're

doing active tests against your infrastructure for it so within reason it's not based on the time objective so if you do have your own internal Incident Response Team it may be best suited that when they're not actively in incidents in other words in the preparation phase they're hunting and that's going to keep their skills sharp too so anyway here's the standard forensics talk we'll go ahead and get this out of the way logging you need to do it you need to do it verbose Lee send your logs somewhere else when possible solely on the fact of adversaries are gonna get in they're gonna try to cover their tracks they may do that via overwriting the logs they may do it

using something like SRM which is going to delete the file and then run as or function over it so that it makes it even harder to recover and then obviously logs are gonna make you or break you inventory everything same thing the whole critical security controls number one and number two know what's authorized and unauthorized from the software and the hardware perspective if you're in a Windows environment there's a powershell script called con so that can help you with this basically you just write a for loop so for hosts in this file duquan so this dump it to a file and there you have it and then you could just parse that using regular expression

into whatever tool you want to use but you need to update that inventory frequently it reminds me of that commercial comes on TV every now and then it's the woman that's got her credit score tattooed on her arm and she doesn't want to check her credit because she's afraid the scores gonna it'll affect the score and her tattoo will be null and void don't treat it like a tattoo I mean treat it like face painting you can change it it's going to change take take the time inventory it make sure that you're up to speed you know what's there because I mean so in some organizations things are going to change daily some weekly I would say monthly or

quarterly might be your best interval for this depending on how fast your organization moves if you're in like a small mom-and-pop office I wouldn't worry about this too much but if you cover multiple time zones you might want to consider using UTC for your long time so you're not having to take your shoes off and count on your fingers and toes the time difference you probably do need NTP to make sure that everything is synchronized if you've got like a 30 second lag between hosts that's going to skew any analysis that you do and when you drop this into a time line it's going to skew it further and baselines I can't hammer on baselines enough we're

going to talk a little bit more about that later on some other things with this is notifications who do we notify when do we notify how do we notify because I do a lot of research in the social engineering field if someone clicks a fish reporting that they clicked it to trigger the incident response process of via email is probably not a good idea you should probably just go ahead and assume that it's compromised so you need to look it out-of-band communications things like encrypted texting apps texting walking over there I know for some of us that's a very foreign concept we don't want to get out of our cube but you kind of have

to phone calls work if you want to go a little bit less conventional I mean there's always carrier pigeons you can do soup soup cans on a string if you're savvy with it and work outdoors you might be able to do like smoke signals Morse code if you've got some crypto nerds in your organization various ways you just don't want to necessarily say hey I think I've been compromised via email because depending on the adversary they can be reading every email anyway and when they see that they could very easily just drop it or modify there's so many things that could happen with it just don't even hedge that and then actions to take this is something that I

wish there was a lot more discussion on so when something happens when you click a fish or something weird happens to your computer or the the not Pecha screen comes up or than what a Christ screen comes up what do you want them to do and this comes to a dichotomy that that kind of doesn't bother me within industry oftentimes we hear users are so stupid why can't they understand how to do security blah blah blah but at the same time we as security people don't care about how to interpret HR policy we don't care about how to maintain a ledger for accounting or a journal I really don't care about procurement as long as the people in procurement buy

procure the things that I need to be procured in a timely manner so why should we expect people not in security to do security functions yes they need to be aware of it just as we need to be aware of other things outside of our bubble but we need to actually say explicitly these are the actions you take so do we want them to hibernate the system do we want them to power it off do we want them to lock it reset it do nothing log off I can't tell you the right answer because it depends on your organization of what's coming next so if you're just going to take it and rebuild it from scratch with known good media

and not do a forensics process the answer might just be shut it off or unplug from the network one or the other and also with this you know we've got a factor in that we've got some non-technical employees in our organizations so I always think how would I instruct my mom to do this my mom's not exactly computer illiterate but she's maybe on a first grade reading level when it comes to technology proverbially speaking so I think about it from the perspective of she's sitting at a desk and something happens and the instruction is to remove the network cable if she looks at the back of that computer and everything is black cables everything's getting unplugged or

nothing but if we say unplug the yellow cable and it's yellow cat5 she could do it if she's not confident with that she could just as easily cut it there's no risk of being shocked with that whereas if you cut the power cable you might look like Marv from home alone a little bit but basically you have to understand what what the response is and it depends on what you're going to do next some things that I definitely would say not to do and again these are highly dependent upon your own policy but a few things would be contaminate the Chain of Custody if a user account gets compromised don't go changing the password just disable the

account or set up verbose logging with that setup simav ents depending on what you're gonna let them do if you're gonna let them continue then just increase your logging level if you want to stop them in their tracks just disable the account don't change the password because if you do have to go to court for any sort of litigious activities you have to have that chain of custody or else that person could walk so another example would be you get compromised with ransomware I think it was Sam Sam because you had RDP hanging open on the public internet without a VPN in between so it's just there for the scanning probably on show den and you log in via

the administrator account with a four character password not that this has ever happened at all and then when you have someone come in to do incident response and they say hey could I see the host to poke around and look and they say oh no we deleted the VM it's probably not a good action to take if you actually want to find out what the real problem is so with that again so let's get into the cool stuff so with memory analysis there's a lot of ways that you could actually acquire the memory image in the Windows environment I'm a huge fan of using ftk imager lite it's free which is my favorite price there's a paid version as well also

recall is a good tool for doing that it's built into Google Rapid Response the other cool thing about recall is it used to be a fork of volatility so it shares some aspects of the same source code but as it forked it's been maintained by Google and recall actually also works on Mac so that's something to keep in keep in mind lime that'll work for Linux and there's a mandate an intelligent response as well in terms of assessment volatility and recall or your primary command-line assessment tools from a GUI you've got red line one thing I'll warn you use something before version 1.2 0 they did away with what's called the MRI the malware risk index it

takes forever to load if you're using an old version it's a java application but you get this you get a red circle on the left hand side and basically there's several heuristic checks they do within the script and the more malicious it seems the higher the number so you see something in the 90s it's definitely worth looking into the thing about redline is it's like a 30,000 foot view and you're digging to the core of the earth when you're in volatility or recall so in that regard so with memory analysis you're trying to find out what's going on with the system you can you can spoof network connections you can modify files on the file system but if it's in memory it's

more or less truth so if you want to find out when something was installed and it's saying 1 1 1970 and you're like I don't think that's the case if there's a registry key associated with it there's actually a volatility plug-in called the registry key creation date and when taking the course Rob Lee the course author basically says that to his knowledge there were no adversaries capable of spoofing that yet so I mean acknowledging that it can be spoofed later yes absolutely anything can be I mean it at one point in time someone thought ntlm couldn't be correct I mean at one point we considered our c4 to be safe md5 sha-1 so it's a matter of

time before something like that happens but as of right now and then you can also look at things outside of your system your syslog or your event log that's going to give you indications of what's going on in terms of the processes the execution and then you're able to cohesively look at things with like Network ports you can actually use it to replace a lot of the system kernel stuff because you've got proc Mon proc dump these are all plugins that are already there you don't have to download anything especially if you're using sift they're already there so you don't have to like if you're doing this in a lab environment you don't have to have

sysinternals running that's one less thing and the good thing is some malware that you may play with if it sees sysinternals on there it's not going to execute so you can snap that memory image run the same tools out of the memory and the malware's none the wiser cross-reference of course you can feed it into a timeline you can look at the registry analysis and you can look for malware so lots of things you can do there here's some cooler things you can do so there's actually a Mimi Katz plugin so my social for indication talks that I gave yesterday and that I'm giving tomorrow actually deals with stealing a memory image for the purpose

of using some of these more nefarious scripts like Mimi cats and hash dump but you can also look at the running processes and identify rogue ones via the for processed ones there you've got Malphite mouse this proc that's pretty fun and then you can actually get the processes in two samples so you could actually get that sample uploaded into a virus total or something and actually get some sort of near immediate feedback some other stuff to consider you got prefetch so prefetch partially there's a good tool to deal with that it's a feature came with XP still in use now basically if you have prefetch configured and you don't have some prefetch files when you're conducting

analysis that's a little sketchy but basically all it's going to do is it's going to get the specific data it's kind of like a cache if you will to help you help the applications run more smoothly from the same vein of caches you have the shim cache also known as the application compatibility hash or cache same thing it's going to look for the compatibility issues it's just basically trying to help things execute faster and more efficiently but you're also going to get other things recorded with this that you're not necessarily going to get with say prefetch such as the process execution flag modification times full file path all that fun stuff and that's in the registry as you can see there so

now let's get into some of the more fun things so with this with processes you can always get hashes of your vital stuff and this really this is the true dear blue team part of this so system32 maybe in the Windows environment you or I'm sorry the environment you might want to look at like Etsy bin and s been basically go in there you could do this very simple with like a batch a PowerShell Bosch whatever you want to use Python I mean if you're a glutton for punishment you could probably even write it and see but basically you'll write a for loop for all the files within the directory excluding whatever you want to exclude

get an md5 of it because again we're not doing this for integrity we don't care about how secure the md5 is because we're just using it to check to see if something's been modified we're not trying to store passwords with it so get an md5 you can use md5 deep for that but then there's another thing called fuzzy hashing which is byte by byte hashing accomplished using SS deep and with that get a sample of that for each file as well so when you go through and you're doing analysis you can compare to the md5 s match yes okay move on do they not match okay what changed in the SS deep now you have a better understanding if

what's changed in that file and you can actually ascertain is this something significant is this authorized as a sudden authorized is this collateral damage what's going on same thing with the registry catalog as much of it as possible but you could script it to parse for certain keys whatever you want to do but basically you want to look at those registry key creation dates with the updates with every time you do significant updates to the computer so that whenever something happens you can go back and say well here's the creation date with this so I said we're going to talk about baselines here we are with baselines there are three baseline plugins associated with volatility

process server and Driver so if you have a known good memory image you can compare your probably not so good memory image with this and actually find out what changed so there's two ways we can go about this every time you update any system you can take a memory image if you're a mom-and-pop place with like eight computers and one server this is plausible if you are Walmart good luck so my recommendation for that instead of taking a memory image of every single host taking them have a standard baseline your golden image if you will have one for the entire organization but then utilize overlays for things like HR accounting IT PR whomever and you have

that standard image so that when something happens you have an image to bounce it off of that's going to require duplicate hardware profiles all that fun stuff but nevertheless you can script it as well but you've got a factor in storage because for a memory image if you have 12 gigs of RAM your memory image is going to take up 12 gigs of storage so keep that in mind so with that just dumping into the whole network forensics thing momentarily get peak apps as often as possible retain them as long as possible but again storage is money so we don't have unlimited budget so keep that in mind integrate with your sim net flow is

highly underrated because with it being metadata it's going to tell you things maybe it won't tell you exactly what's going on but it will give you some indication of what's happening if we think back to the whole controversy with NSA collecting metadata I mean they said oh well we're not collecting actual data so it's okay well let's think about what NetFlow does right so we're talking about who communicated with whom at what time for how long how much data was passed fundamentally that's what NetFlow is going to give you so okay you can see that this host was compromised was communicating with this compromised host there was this much data sent there's a good chance data exfiltration occurred

also you can look at it within any sort of analyzer or your sim to actually see spikes so hmm three o'clock in the morning we're not doing updates no one's in the office from 10:00 p.m. to 7:00 a.m. something happened at 3:00 what's going on here so that's giving you something to investigate something to dig deeper with your logs same thing they should be feeding into your sim so that you can get that information some sims will do peak apps for you automatically which that's pretty awesome if you can get that to work and then some integrate with vulnerability data so you're basically looking at this from all layers on top of each other in opportunities where you

can carve the peak apps out as much as possible you can do it manually with Wireshark or network miner will do it a little bit more scripted for you not even scripted it's just point and click so one thing that I always like to do with it when doing the demonstration I'm not doing that demonstration today just because it's not one to play in my VM environment but anyway I'll open up a peek app and there's a file transfer over SMB that is what the peak app is and I always show hey I don't have putty on the system I don't have putty installed it's not there but it carved out putty and you can just double click

on it and all of a sudden putty pops up so you've got the executable that was transferred over SMB other things you can do it makes it really easy to trace what happened like in terms of which hosts communicated with which the data streams the sessions all that fun stuff you can with file so you can also get the hashes or you can even open them you can open the file you can look at the path you can get the hash so that works out really well with uploading things into virustotal and then creating your own threat intelligence feeds so that's pretty much that so just to kind of wind down on this so Brian Austin and I created through the

hacking glass it's a mentorship platform not with peer list anymore but I've not updated this slide but basically we understand that academia serves a purpose and certifications serve a different purpose but we still have employers looking for people with 14 years of aetherium experience hey I just saw it on indeed but anyway because of that we want to help people get experience with things outside of certification and formal education so basically we're trying to pair people with mentors or pair of people with mentees and look at things such as hardening system attack so like the pen testing side entering incident response and analysis so and at some point will have a range built that will actually have it to

where you will go in someone will go in and harden a system then the person doing the attack phase will attack that same system or systems there's varying levels of difficulty while someone's monitoring it they'll pass that off to incident response and at the end we'll all hop on probably a WebEx or something and talk about it at the very end so that'll provide the pen tester the opportunity to tell the guy that hardened it or the gal or whomever hardened it hey I really like that you set this thing here this way I couldn't get in I was planning on trying to make it in this way but because you had this setting in place I couldn't do it or hey

you know I know what you were trying to do with this but you left this hangin wide open and I walk right in the front door whatever something to that effect so to sign up for it just go to the Twitter or the Facebook we've got links to the mailing list that's actually what we're using right now for the sign up that's on we're using MailChimp with that it's gdpr enabled so we're pretty good with that I guess future speaking engagements I'll be in Atlanta tomorrow that NOLA con and then a few other things so I like to give away free stuff so if anybody wants to come to hacker halted in Atlanta in October here is a

coupon code to get free tickets tickets are normally $199 if you want 15% off of any ec-council training there's a coupon code for that they're offering everything so see eh CH fi e si si el PTC C so whatever you want to do good conference we've not announced any speakers yet the CFP just closed but I will say that the keynotes the two that I can remember off the top of my head are KCl is from bugcrowd and Paul acid Orion from security weekly so we've got that going on as well and if you want to attend my full-day social engineering and open source intelligence training I'll be giving it on June 7th layer 8 security

conference I forgot the 8th in the slide that's what happens when you fly in late at night and you're trying to get to bed didn't make it here on time and you're just like drugging through it but anyway there's where you sign up if you the promo code at the bottom you'll get $50 off so I think the sticker price on it's $4.99 that'll get you for like 449 but either way it's a full-day course covers both go cent and social engineering and there are labs associated with it so with that being said any questions

yeah probably see me afterwards I'll give you my card just shoot me an email or hit me on LinkedIn or Twitter and I'll hook you up any other questions concerns compliance grievances yes there and that's a possibility but even if with a collision with md5 you're gonna back it up with the SS deep so you're not relying on it the same if we were talking about passwords I would be 100% onboard with it and yes but if you feel more secure going with like sha-1 or shot 256 whatever you're more comfortable with the only thing you have to think about when you're hashing this is the time to hash so if you're using like you know Shaw it's going to take a

little bit longer so but it's the time coefficient it's such a new possibility that honestly exactly so right okay so so I mean and again you could have that engineered collision and yes it's going to cause a headache you may be you know wasting a little bit of time with it but at the same time you just have to look at the time you would waste with that comparing it to the md5 that's the engineered collision versus the SS deep because they're not going to get the SS deep collision so you'll see that and be like okay which would be an indication of something nefarious in itself or you know do you want to spend more time

using Shaw you know whether you want to use one 256 or 512 whichever any other questions concerns complaints grievances if not then that's all folks [Applause]

thank you under budget and ahead of schedule right