Effective DFIR Triage Techniques to Detect Modern Rootkits

threats in Windows Linux and Mac memory tux abstract is going to be the wealth of data available to incident response handlers during breach investigations is often overwhelming both to Junior and senior senior analysts alike depending on the I.T maturity of the victim organization the state can range from days to months of forensic data acquired from hard drives of volatile memory Network sensors AV EDR engine Sims and Beyond effectively efficiently locating signs of malware intrusions and such a large data set large data set requires an analyst to possess techniques that lead to Quick wins and avoid falling down rabbit holes this presentation a walkthrough of effective dfir techniques will be showcased it gets Windows root kits that have been discovered in the wild through a combination Target file system and memory analysis attendees will see precisely where the most actionable artifacts reside and how to detect such malware in an automated fashion and they will be able to apply these techniques in the field to detect threats throughout environment State protect get it for Andrew y'all thank you yeah so thank you all for coming out to the talk that was rescheduled multiple times uh I live outside New Orleans and tried to fly here yesterday and it took 27 hours to get here between all the cancellations and stuff so that's why that's why I got here a little bit late um but again thank you all for sticking sticking out and coming to this slot so yeah today I'm going to be walking through a project that myself and one of my co-workers at velixity did recently to find the most effective ways to find root kits in memory samples and data sets that we were looking at we do a lot of incident response and then myself in Austin who who I did the project with or on the research team so we do all the coding and research to make sure that our services team can find data as quick as possible and our motivation for this is if you're in the room then you know root kits are a serious threat but if you don't follow the space closely it's something that we've noticed over the last two to three years is that there's been a significant rise in the usage of Windows root kits by apt groups so we work on cases that with clients that are targeted by nation states targeted by very motivated actors and for the years before that it was mostly user land stuff so if you look at the stuff we published if you looked at work from other security researchers forensics researchers there was a lot of push towards powershelldetections.net detections and all of that but we in the back of our mind we kept seeing root kits over and over and if you look at reports from other vendors there was these new root kits as well and for the years before that there wasn't a whole lot of root kids which is what led to the userland malware being pushed out so hard if you haven't heard of driver signing enforcement before you're going to hear it a lot today this was one of the main reasons that rootkits kind of fell off a cliff on Windows but as we'll see the attackers figured out how to get their malware loaded anyway which brought us to where we are now so our goals for the research were to develop effective and scalable triage techniques effective meaning when you use these techniques on a memory sample or some files from disk you don't get a thousand false positives or out of 10 systems you get six false positives and one true positives that doesn't one true positive that doesn't work when you're trying to really work at scale when investigations involve tens hundreds thousands of machines and then scalable like I mentioned just looking at one memory sample one disk image that's not where we try to focus our effort now because it doesn't work out anyway to start this we needed to do a deep study of rootkits from the last couple years we see them a lot in our own investigations but as you probably know that means you look at them you reverse them and then you forget it 10 minutes later or it's a note somewhere you don't look at again so we wanted to do a really systematic study of root kits that we've seen we have a lot of internal memory samples and malware samples and then we also have just virus total and Connections in the threat Intel world so that we can get samples that are in other vendors reports so we did a very thorough study over probably 12 20 root kits somewhere in there and we wanted to see how they were loading on systems with driver signing enforcement enabled it's a little bit puzzling once you'll learn about it in a couple slides and then with patch guard which you'll also see a lot today in this talk patchguard stops a lot of the old ways that rootkits infected systems so we knew we could detect those old ways while at the same time we knew they weren't really being used unless you had like an old old Windows 7 or Windows XP system so we wanted to see which of our techniques where we which malware techniques were we still detecting on Windows 10 and which new techniques were being used and as far as where we looked at for our investigation our sources we the first place is where malware operates which is in memory if you're familiar with volatility that's probably not a big shock that we went there first but that's truly where malware operates and then the second thing we looked at which is besides memory our favorite forensic Source are the event logs because those can have historical records going back way back in time if you're not familiar with memory forensics just real quick the idea is what I'm going to call basically all modern payloads at this point any payload you're going to see in a real attack in the real world is going to be memory only so it's not going to write anything to disk anything that's on the network is going to be encrypted so if you walk up to that machine and you pull the plug or you say it's like an ec2 instance and you just use the apis to get a disk image out the data that you actually need is in memory and if you don't look in memory you're not going to find the malware payloads in many cases you'll find absolutely nothing at the bottom of the slide here this was a report actually a really nice report from Microsoft yes is there a question so so root kits and the stuff I'm talking about today specifically enter kernel mode so a lot of the other things you might be familiar with like Cobalt strike Powershell Empire interpreter like all the and then all the private ones used by direct groups when if you're just talking to a C2 server something like that normally that's just in user land that's just in process memory now you can inject code into a very privileged process like service host lsas whatever but you're still in process memory where once you're in kernel mode it's totally different as we'll see you have full access to the hardware to the other every process on the machine it's a totally different thing and you have to really change your approach on detection as well and then yes right and you it can persist on disk as we'll see with like a service but it's the the differentiator between root kits and everything else is that it's in ring zero and that has code there some some summer kits don't persist off of this because especially if you're in a Windows like active directory environment which most organizations are um depending on your tooling there's not a whole it's actually kind of uh it's not a great idea to have persistence on disk on critical servers so instead what you can do is maintain persistent on like a machine in accounting that no one cares about also you have a lot of machines and environments with like no security software that people just forget about so we'll see persistence on those but then there's no persistence on the critical servers because you have credentials anyway so if the machine reboots you just reinfect it through the same thing you did before there's no reason to leave files on disk there so that's what we see a lot and it's also it's not really the point of this presentation but it's also why when we talk more generally about how to approach large-scale IR it's really important to find to be able to trace through the network all the machines because if you want to actually find the persistent stuff it's going to be on the machine with no security software on it that's in a closet somewhere that no one knows about so that's what we see a lot but yeah I should probably started to find that before so thank you for the question Rook hits here mean we're in ring zero we have the most privileged access to the system and uh at the bottom of the slide here it was just another example of the power of memory only malware Microsoft basically burned this Australian security company that was selling uh zero days and tool sets and like tool sets to use those zero days and they were targeting journalists and other people so Microsoft basically burned the company to the ground they took all their malware wrote some awesome technical reports on it the malware was called core lump and you can see what I have highlighted and read here from the report is that all of the payloads that actually matter the key loggers the screenshots Expo trading files those are all memory only so again if you're not looking in memory you're not going to find what you're looking for also I have at the bottom if you haven't seen one of my talks before you'll see slides with numbers and brackets the very last slide is all the URLs so rather than flood the slides with URLs they're all on one slide at the end so that's the memory forensics part the other part is event logs these are an amazing resource if they are configured correctly and in a lot of the environments that we work they're centralized so if you know the event IDs to look for if those events are being fire or firing on the machines you care about it's as easy as querying them out and now you found all the malware you found every infected machine and event logs and as we'll see event logs and memory forensics play really nicely together because if we have a very solid event ID that tells us that a root Kit's on a machine but what are we going to do after that go take a memory sample and really deep dive that system it also prevents us from having to take memory samples from thousands of machines in the first place when we can filter down to a much smaller set by using the event logs in a smart way and then the other thing is if you're familiar with old style malware old style tool kits for like Windows XP Server 2003 yes you can wipe the event logs on those it's a pain but starting with Vista and all the way through Windows 11 by default you have something like 40 50 event logs and then you have a couple hundred by the time you have all the other software installed so it's just not feasible at that point to wipe out all the records there's overlapping data there's data that the attackers don't know about unless they're like forensics experts which most of them aren't so it's very we don't see that very often it's not really an issue anymore even on single machines and then like I said in your corporate world you probably have centralized logs especially from those critical servers so with our two sources let's talk about how root kits actually load and what the type of malware we're talking about is so we need to create a service this is the Microsoft recommended and provided way for loading a kernel driver so remember that driver is going to load into the kernel ring zero access with access to everything so you create a service of the appropriate type this is going to create some entries within the registry so it'll have the path to the driver the name of it and so on and it will also have when it starts this does create a records in the event logs but I just wanted to point out that I'm not going to be talking about those because they're super noisy there's hundreds of services on a single machine it's even if you have a white list it's very hard to try to narrow that down so we're not going to focus on that from a triage perspective but I'm just pointing it out so no one asked about it later so we have our drivers that we can load through a service before driver signing enforcement which I'm just going to call DSE from now on because it takes a long time to say before DSE was there in Windows 10 rootkits could just with admin access create a service point the point it out to their rootkit file and now they can load into ring zero whenever they want the other thing to be aware of is you might be saying well at that point we find the rootkit right away that is true but most of the root kits especially the good ones were just loaders so the same way that user land loaders are just small pieces of code that download files over the internet and then execute something the root kits work the same way they're going to download those kernel drivers over the Internet or they'll be packed and encrypted inside that smaller driver it's not as simple as just throwing something in Ida Pro okay I'm pretty good at talking over things but that would have been tough um so with driver signing enforcement it broke all of those abilities for root kits the first thing that happens is the drivers have to be signed uh before Windows 10 it was not enforced and to get a driver signed for Windows 10 and this is something that we dealt with at flexity and it's it's a pain there's a quite the learning curve to it you have to use this entire software setup from Microsoft you have to get an EV certificate which is super difficult to get they verify Way Beyond what they used to before and then to get the driver signed you have to actually submit your driver file to Microsoft so as a malware developer you do all your anti-forensics tricks and everything and then you send a copy of the file to Microsoft like come on that's not going to work so this is great from a security perspective except as we'll see in a couple slides malware authors figure out how to get around it in quite a few cases but this is a significant hurdle this was the reason we didn't see great advances in Windows root kits for quite a few years because the malware authors weren't able to break it and if you're wondering why besides or I guess the main reason why Microsoft was doing this is highlighted at the bottom they were just getting thrashed from stolen signing keys so they would give out signing keys to developers those developers would get hacked now there's legitimate malware if you look on virus total a lot of malware reports there's something signed that a lot of the AVS just skip it or they don't look at it as hard and there's been some pretty devastating attacks with this as far as I know the most devastating one was stuxnet it the that malware was all signed including its rootkit driver from stolen certificates remember there was no cross signing then so if you had that stolen certificate you could sign the executable and then this is not just something from 10 15 years ago these are two articles with Microsoft still signing rootkits and then very famously a couple months ago six months ago now Nvidia had source code stolen with the certificates yes software manufacturers are now technically kind of your own group kit to make sure you're not stealing your product through your own drivers we do see it we do see it a lot Microsoft still signs those they do show up in memory a lot we end up the first time I kind of know what like semantic and crowdstrike and all those look like in memory now from looking at it so much but the first time you see like an EDR you're like wow this is malware and then you're like oh no it's just it falls back to some driver like um I think it's crowdstrike makes uh makes files that are named exactly as malware would make them like kernel 32 where the first e is a three and a bunch of craziness and you're like come on man this is like evil this is and then you trace it back and it's just crowdstrike stuff and whatever and there's a bunch of examples of that it's not just them um but yeah it's a real pain when you have when you have that on a system so but yeah so stolen certificates led to many devastating attacks so Microsoft wanted to stop that that's where the cross signing came from but of course attackers still want to be able to load their root kits on system and get that ring zero access so there was this headline here there's a I guess we call an attack type or class called bring your own vulnerable driver I'm not going to say that over and over again I'm just going to say vulnerable driver from now on the title here is probably a little bit uh overblown but it looks really good on the slide so they're breaking Windows the root kits are loading and this is recent right this is from this summer and then I think this second one was from October so since I even submitted the call for papers for this conference another apt group was found using these attacks to get drivers loaded and the idea is which we'll look at the details in a couple of slides is by using a vulnerable driver that's signed you can basically piggyback off of it to get into ring zero yourself so you don't have to submit your own driver and all that to Microsoft and this is a pretty prolific way to get into the kernel this is an amazing article the the screenshots here come from an amazing article from rapid7 where it was published in December 2021 so it doesn't have any of the root kits from this year which you just saw there's quite a few of those as well but they profiled all of the main apt real root kits used in real targeted attacks that use vulnerable drivers so I definitely ran recommend reading that if you're new to this vulnerability class the article is really nice and so you can see there's quite a few apt groups quite a few Brew kits that are using these vulnerable drivers to still get access to Kernel mode even though they can't get their own drivers signed now I want to quickly Define vulnerable because it is not just it's a you know drivers or C C plus plus it's not just buffer overflows integer overflows things like that drivers do have to provide access from processes to be able to talk to the hardware your Nvidia manager where you can set all the settings on your video card and all of that that program that you're clicking in is running as a process and it has to be able to tell the video card hey do these things and set these parameters So that obviously needs to go through the driver to get there same thing with your network card and every other piece of Hardware that you have and then there's a bunch of these just built into Windows as well so with that access if the driver doesn't lock down which processes can make those calls and it doesn't lock down what regions of memory or which Hardware registers can be written to by those calls then malware can just read and write directly to the kernel so this is what we see all the malware you saw on the last slide all the stuff from this year they package a vulnerable driver with it they know how to exploit that driver so from a a process they can read and write kernel memory and then from there there's two approaches they can take the first one is to directly manipulate kernel code and data but I will say that many of the targets that you would want for this are protected by patchguard which you'll see all the details of that shortly and this is really hard to do if you've ever written Shell Code or anything like that where you're not using visual studio where you're piecing together assembly code imagine trying to do that for a Chrono rootkit it's completely miserable so there's a couple root kits that do this for very very specific operations like hooking functions but I would say 95 plus percent of the root kits that we looked at from this research use approach two which is they're going to disable the driver signing enforcement to then