Stacked and Hacked: Crafting the Ultimate COTS Incident Response Arsenal - Art Ocain

all right good okay

cool let's try not to knock this off all right this is an interesting audio situation can everybody hear me all right cool great so I'm talking today on incident response and the tooling behind it it can be kind of uh polarizing and uh confrontational uh because everybody has their favorite tools uh everybody has things that they've swear sworn by and a lot of that is driven by the risks in your organization and what your infrastructure is like and what kind of analytics you need to gather to do effective IR uh and and really what kind of attacks you're seeing uh but there are some parties of people they kind of hold on some traditionalist forensic

approaches to IR that really isn't needed these days because they're Gathering stuff that they'll never use unless they're Prosecuting so there's uh it's kind of a an interesting uh a play between um function and uh you know do I want a pretty forensic report at the end anyway so everyone will not agree with some of the things that are are common um tooling that we're using in IR now uh you know and you definitely probably are using tools that I'm not going to talk about which is fine um you know there are thousands of tools plenty of Open Source things you can do everything you could possibly want in an IR with a lot of tools or you

could just like buy Sentinel one and use it and probably have an effective outcome so uh there's a lot of different ways to skin the cat I'm a nerd I have a lot of engineering EXP experience a lot of assist networking Cloud admin experience uh but uh you know Last 5 Years have been really deep and incident response on the front lines we have dedicated incident response team at ariam which is where I work and they are uh you know handling between five and 10 incidents at any time so you know there might be a couple guys working on one incident might have 40 on another incident really depends on the type and scale of the

incident we're getting called in from uh the insurance Channel and uh you know so they're people that we're not actively managing they're people we've never even heard of brands that are calling us and some of them are name brands our Fortune 500 companies are are very large companies and we'll get a call to do their ir and you know we have no previous log information uh no history no understanding of their infrastructure we're walking in blind to solve a problem so raise your hand if you have defined priorities for your incident response team or it you know some of you are in incident response do you know what the priorities of the business are if so

raise your hand okay and the reason why this is important is because it's really is going to shape your tool selection you know my in my experience almost every IR I've been on the uh the focus has been to minimize business Interruption get the company back up and running so they're making money minimize bad press and stop that uh that leakage of their intellectual property and and uh uh you know private data of their customers that's usually the goal um with the incidents that I'm on prosecution and attribution is really less important it's it's uh usually you know most of the time we're getting a call when it's ransomware or some other bad destructive cyber incident that is is kind of a

business ending event and they're focused on getting up and making money uh so if you're in a highly regulated at ated environment or in law enforcement more forensics is probably going to be important to you so those of you in IR do you have a defined IR stack and do you have processes and procedures around that can you raise your hand if you do you know these are the 10 tools we're using these are the processes we're going to use everybody knows how to use the tools that kind of thing and again this this will vary depending on your business and what your needs are if you have infrastructure that you need to gather a lot of logs

from then you're going to need a SIM or a sore you're going to need you know a at the very minimum some log collection massive CIS log server but something to give you something to search from but uh if you're if you have a real like a remote infrastructure where everybody's you know you have no on-prem environment you may not need that you know it may be that an xdr and EDR is more appropriate for for you so when you're determining your priorities for your strategy um I kind of put what it usually looks like for me on the screen this could vary but you know minimizing Financial impact and business Interruption is usually number

one and and and then bad press and avoiding litigation that kind of risk that that that negative risk to the organization are my my top priorities and the rest of it is just extras it's like okay yeah that rtion and prosecution that's cool to have but uh you know really less important and a lot of companies that I work with are willing to throw away forensic data in order to get back up and running and usually I caution them against that and say hey we want to keep the forensic data in case there's a reason to have it later in case we need to do a deeper forensic investigation later but that does not happen at the beginning that's

I'm saving their data and their forensic information so I can do an investigation later after I get them up and running um you know so it's it's usually I'm driven by these priorities now some of you are are more forensic minded and you may reverse this uh you you may have different priorities they driving your organization but what I would caution you that are n to do is really make sure your strategy and your team and your tool selection is in alignment with what the business needs if the business doesn't care about the forensic side of it then don't invest heavily in it if you need a deep forensic investigation you can pull in a forensic partner and

have them do that without investing heavily in in a forensic tool stack so there's an incident response Challenge and it's usually speed it's usually getting back up and running as fast as possible minimizing that that Financial impact but one of the other challenges is adapting and being resilient and and uh reducing your future risk and a lot of organizations uh you know will just get back up and running kind kind of how they were before the incident and then they'll just get popped again because they have done nothing to improve where they are so when you're in an incident Gathering that information so that you can make an informed after action lesson learned type uh response is is

critical and uh organizations fear these days business Interruption breach of data encryption of systems um you know losing availability of of public facing stuff uh and losing their IP bad front page stuff all right so before I dive into tools let me just talk about the process here uh the incident response process and and there are others if you work for mandant or something like that they have their own uh process and and and procedure that they go through but uh the nist 861 insurent response cycle combines a lot of the things that are basically granular now in the Sans incident response process but there's a lot more than uh uh just the investigation there's you know

containment eradication of the threat recovery is something that gets skipped in a lot of ir thinking a lot of people just aren't focused on recovery and then okay great I I I've stopped the attack and then the business is still down for the next 3 weeks you know uh recovery is still something we have to Tool and plan around and if you notice that detection analysis and identification phases of these that's really your sock analyst your simor xdr products your threat intelligence your friends of Investigations that's really where that is although obviously sock analysts are going to be involved in that containment and eradication as well um but it is uh most of our investment that we're doing

in security only aligns with one part of incident response it's really this initial figuring out what's happening and kind of you know uh detecting that attack so if we look at kind of how the Investments go with with organizations usually everybody's spending all their tooling on identify and protect and then kind of it starts petering off and uh that detect respond recovery uh recover is usually a really light investment with organizations they're like we have VH we're good to go if there's an incident and which you know may be helpful but it it's really the with the frequency of attacks you know most organizations are going to be hacked you know once every three years with a

significant incident and you know the fact that we are

underinvestigated process and this is really how we approach it at arum is instead of doing it linearly most of these things happen at once we'll have somebody working on an investigation figuring out okay what are my ioc's you know what what happened what's my patient zero um you know what are the the vectors of attack uh what persistence mechanisms are in the environment now what back doors have they put in things like that that's an investigation that has to happen but it need if you're waiting for that investigation to happen before going on with the rest of your process you're stalling your whole incident you're stretching out your incident life cycle you know I will do while in forensics is

doing an investigation we will we are focusing on containment of any as soon as a new ioc is is deter is found soon as we're understanding really what the ttps the attacker using are you know and this is developing as we're doing the investigation as soon as that happens we're starting to contain on that and we're starting to put guard rails around there that and then we're starting recovery from day one also we'll set up a quarantine environment in order to start recovering systems into a safe environment even before we have a handle on the fire that's happening in the production environment we don't want to wait we want when when the fire is put

out we want to be able to just flip over to production and get them back up and running so we start recovery from from the beginning so we have a kind of a unique uh way to do that but uh I I would say that my guidance for any IR team is don't get stuck in linear process thinking uh you know each of these steps is critical you need to have them but they can happen at once you just need to plan for that and structure your plan and your your tooling around that all right so I mentioned some of the the tooling that's common in incident response Sim IDs IPS a lot of people aren't really Gathering

a lot of network data that's fine cuz most attacks are happening at the uh at the host level but but uh you know the network layer can a lot of times that is how you're identifying those OT attacks those iot attacks if you don't have eyes on that Network layer a lot of times you're missing that stuff and your network layer is also how you're identifying that C2 traffic so you're you you'll recognize that there are 20 machines calling out to a t C2 server from that firewall traffic that's how you're going to find that out so uh it's important to have ways to gather that forensic collectors um like Velociraptor very common uh you can replace

Velociraptor with a bunch of scripts um so just to gather those artifacts uh and logs so there's a lot of ways to skin the cat but uh and then there's EDR that's the most common these days and in a lot of professional IR companies we really lead with EDR just deploy EDR gather artifacts through EDR find out who is affected uh find out the scope of impact in things through EDR and then there's log analysis tools so these are all open open source that I've listed here but uh really Gathering the the logs those artifacts that you're Gathering through EDR or through Velociraptor or whatever tool you're using even the Sim and then shoving it

through log analysis tools putting together a timeline and and figuring out really what what happens and where to go network analysis tools are very important because a lot of us don't have the visibility on the network that we should and a lot of us in security may not be completely like Network Geeks uh it's great when we have those on our staff but when we don't it's great to have tools that we can just pump those Network logs in and be like tell me where the fire is and the tools will help uh so dis forensics you know there's a lot out there you know ftk the sleuth kit autopsy uh end case but uh

like I said those are actually used fairly rarely now in incident response so uh you know it it's not that often that I need to do a deep disc investigation on something and figure out you know what was really going on on this machine um usually EDR or other tools are giving me the artifacts that I need for that information so uh dis forensics if I need that to happen usually I'll pull in a company that that that's their specialty I'll I'll I'll you know maintain chain of custody and send them the drives and uh and have them deal with it all right so I pulled in some research from some partners of ours and

some people that we love working with and and uh one of them is trusted SE Justin Vio um is in leads incident response at trusted SE and really says without logs it's hard for us to have any visibility so those of you who are leading in incident response team right now make sure that you're Gathering the Telemetry that you need to across your organization in order to be effective uh so the fundamental piece of The Trusted sex looking for ioc's timeline and scope of impact they want to know what was affected you know how would the impact to the company what the ioc's are and what the timeline of the attack really was they use an offensive

offensive mindset which I love to help understand the attack they're using Blood Hound things like that to really see kind of like the attacker would see you know enumerate what what what is there and see okay uh if we were able to do this the attacker was probably able to do this as well and understanding how tools like Cobalt strike work is really critical too because uh you know in a lot of these situations you're using the attacker is using offensive tools so understanding how these work is going to give you a leg up and they say that really they they create a spreadsheet of Doom it's you know an Excel spreadsheet that really has this information this is what

they're trying to gather an incident summary with all the compromised systems ioc's compromised accounts email addresses timeline key file and hash information this is their work product that they're getting together so when we're talking talk about the tools this is what they're using um you know they're all open source you know they're not buying a lot of commercial Tools in order to be effective at instent response so uh this is one thing that that I have a lot of arguments with guys even on my own team they want me to invest in huge tooling uh in order to to be effective at instant response you really don't have to um the the reason to spend money on

commercial off the the Shelf software is for Speed to Value it's okay I need to be up and running with an effective team now and I need to automate as much as possible now and I can't screw around with open source tools that's when to buy stuff um if you have luxury of time and you have a talented team play around with some open source tools uh I find that even when you have a lot of uh software investment you're still using some open source tools uh to to manage specific logs let's say you have stuff on all the endpoints but you're not Gathering Network stuff you're going to grab those Network logs and then use an

open source Tool uh you know in order to analyze those something like security onion on those logs and uh you know obviously log collection and scale is important so using an elastic cluster as a Sim or using an Enterprise solution like Splunk can really help you with ir you know most of your organizations because of your sock already have something like this in place um but uh if you don't and you're in an incident sometimes it makes sense to spin something up last minute even if it's just a Cy log server Point all your firewalls to it Point your VMware environment to it Etc and start Gathering that that information so that in the midst of the incident you can

have something to work with now binize is a software company and they have a d d for platform so if you're looking for an off-the-shelf platform to manage incidents they're amazing and I would highly recommend using them um they will integrate with you know your firewall stuff your your sim your uh EDR your xdr platforms and pull all of that in to manage the incident and tell the story of the incident they'll gather that artifacts help you you're able to threat hunt right through it you're able to be super effective using this platform does cost money so if you're in an environment working for a company that has a budget for ir and effective tooling baliz is something to to

consider as far as an off the-shelf tool they can make you effective if you have Sentinel one or crowd strike and and you need something to be able to to Really analyze the artifacts that it can gather you know integrated with miniz now Jason Rebels some of you have seen his stuff on on um LinkedIn teach me Cyber he's always posting about um he's the ceso at uh corvis insurance I met him doing incident response um and he uh is the co-founder of MOX 5 which is an in incident response uh advisory firm and he recommends using you know breor lab scripts so uh you know really open source scripts to gather artifacts and imp parse those artifacts with open

source tools you know he worked in mandiant crypsis like he uh you know has a big Rich incident response background and he leans towards open source uh rather than paying for tools he emphasizes that the documentation is actually more important than a lot of the stuff we do on incident response um you know a lot of us get get really deep in the weeds as far as uh what the thread actor was doing and trying to analyze the malware and stuff but uh documenting what you're doing and what changes you're making to the environment can be even more important in the end you need to make sure this change management in place is what he says says now threat Locker many

of you have probably not heard of threat Locker threat Locker is a uh is an MSP tool for endpoint security really they're they sell through msps and uh they have some amazing functionality including Network control which basically puts a host firewall uh but they have application control storage control Etc you're able to lock down a machine with threat locker and it's very inexpensive so as soon as I saw threat locker and and started using using threat Locker I was like hey why don't we use this in instant response threat Locker had never thought of that it wasn't something that you know they were just focusing on endpoint security but not really thinking about an instant

response tool now I'm effectively able to deploy threat locker and stop any incident in tracks it will stop any scripts from executing Powershell um any me memory resonant any any really any execution it can stop in its tracks and then any kind of uh network uh Telemetry it's Gathering uh storage access it's Gathering and you can stop in dead in its tracks any storage access so uh being able to use a tool like this for incident response it's an off-the-shelf commercial Tool uh you know and they were they're delighted that that we're using it for incident response to me it's a go-to if you need to stop an incident dead in its tracks deploy threat Locker across all your

hosts you're really effectively you know C frustrating the attacker they're unable to do anything now the way threat Locker working they're taking if you turn on network control and threat Locker they're Gathering all these records they're running everything through virus total um you know and they're mapping things using their cyber Ops platform there's a lot going on behind the scenes because it is a security tool it's not EDR so EDR is using your behavior to detect whether there's a threat uh this doesn't care about your behavior it is going to stop everything from running unless you purposely allow it to run so in the midst of an incident if you want if you need if you just need to get a

handle on things and stop things from executing uh you know you're able to use it if you have the luxury of getting threat Locker ahead of time before the incident they're able to put it in learning mode so it will create a Baseline and then block anything that's outside of that Baseline uh so it is you know really effective in that in that way so we use threat Locker on all servers and workstations in an incident but then we'll use singulars cloud defer uh product for cloud resources and that's super effective also uh set it up for you know join it to uh your 365 environment your Azure environment your gcp environment your AWS environment it

will gather all the artifacts and it's live connected so it's seeing any thread as it's happening uh it it will you know alert you way before you'll get an alert from Azure security Center or from you know from AWS so uh you know it is super effective it's using Aid driven you know um uh threat hunting and it's constantly doing this so uh it's it's able to identify a uh threat and attack in minutes and it'll give you a fix action it'll say you know uh basically open your Powershell for Azure and and run these five commands and it'll nuke these storage keys or rotate these creds or or suspend this IM account or or shut down

this resource it's giving you these these fix actions to use so these are kind of my off-the-shelf stack it's threat locker and singular and it's pretty inexpensive and it allows me to stop anything in the cloud and anything that is on Prem on hosts and servers what's missing okay so any tooling you you pick there's some trade-offs there's stuff missing I'm missing that Network layer with this when I when somebody calls calls and says hey we're on fire I don't and I don't know nothing about their organization I will go in and deploy threat locker and get singular set up um and then I'm starting to worry about Network logs and then I'll start to gather those manually um you know

hopefully they have a SIM that I can use if not then you know set up a CIS log server or something grab that but missing from this stack is that that Network stuff it's missing the OT and the iot monitoring if you have a heavy o and iot environment you're probably going to want to engage dragos or something like that in order to pull in that that right Telemetry for that and start watching that an identity side you know a lot of you are using you know OCTA Azure ad things like that you want to pull those logs in from somewhere into into a Sim so that you can do analysis on that as well so those things are missing from my

stack this is my go-to two products that I'll deploy and then really I'll adjust depending on what the incident is what the customer's environment's like and you know add tools or remove tools as needed uh the weakness of this stack is that I'll have to pull in some open source tools to look at Network logs things like that because I'm not investing in that in that stack I'm I'm I pick tooling that I'm going to be using on 90% of incidents um and that's my default now understanding that it's important to note that any team is going to have processes for using some of these other tools and understanding that you're going to be using hey what we have you

know the we have three offices that have poo alos how do we gather those logs and start using that that data and having processes around that um understanding that is key to your success with ir now threat Locker allows us to ring fence applications so we can say these applications are allowed to talk to these things which are trusted or that you think are trusted but nothing else so people can't use word to go and launch Powershell or can't can't use word to to uh you know load a dll and start doing other stuff uh we are you know using that to ring fence what you're allowed to use um and then it it's able to block privilege escalation

access to storage Etc that's that's really what makes it a super effective solution thre Locker Ops also detects and alerts so you don't have to sit there and actively watch it all the time it's cool if your sock is uh but you you can really set it up to alert you and set up web hooks to to do what you need to do it also threat Locker also supports uh full lockdown mode so uh if you're starting an incident you can deploy it and lock just lock the environment down nothing will run you can't open word you can't open notepad nothing will run and in in a bad incident you may want to do that you know you may want to just

stop the presses on everything and then open up segments and so you're using that as a tool to create containment and contain the threat in your network now Cloud defer allows us to identify those threats a lot of times you know with normal sock processes with our normal Sim and sore we're missing some of these uh you know Slow and Low Cloud movements and and threats um so Cloud defer really does that threat hunting for us and adds that and it's Gathering that timeline for us too so uh that timeline's super critical to understand when did this incident start you know what happened and how did it progress uh it's really effective at doing this

so those of you who are um you know may may consider using something like threat Locker something that's kind of important to know is you can call threat Locker now they will answer the phone in less than a minute their SLA is less than a minute their average right now is 26 seconds on answering the phone their Engineers will help you set up a tenant and install threat Locker on the phone and get it deployed to your environment and they're not charging you money for this they're they will help you get this running and then they'll talk about a contract you know and talk about like hey how about if you buy this and let's

well let's answer your incident as a as a proof of concept but then um you know we'll worry about selling it later that's a super pro for being able to pull in a partner like that a uh a company like that so keep them in your mind for when that bad thing happens but it's definitely better if you get it in ahead of time now Cloud defer um they charge per um resource in the cloud so if you have an S3 bucket that's a resource if you have a VM that's a resource if you're running container services each container is going to be a resource you know uh wff is going to be a resource so they're charging you per

thing that they they're monitor ing so uh but as far as off-the-shelf products um those are my go-to they used to be EDR was my go-to just deploy EDR and then go from there um EDR is great and we're still using that a lot but um you know this allows me to to create a a safe containment in the on-prem environment so that I can start recovering right away now the speed to us is the most critical part being able to respond quickly is really our brand and being able to do that so that's why we've chosen those tools or we could improve our stack with an xdr that's able to pull in that Network stuff able to pull in that

identity stuff and then integrate with threat locker and Cloud defer and pull in all that to a correlated system but uh you know let's face it with an OT based attack you're going to pull in probably an OT specialist something like dragos uh to handle that and uh you know so if you're heavy OT Environ definitely invest in in that ahead of time all right if anybody has any questions feel free to come up and ask or or find me as I'm Milling around like I said you know everybody has different tools that they love you know and depending on your environment you may need more forensics you may need uh a different stack but uh

it's important that you align with really the business need needs in order to get back up and running safely um so that they're uh you know you're able to minimize business Interruption and financial impact and and go from there thank [Applause]

you