How EDRs See Everything (Until They Don’t) - Jacob Kalat - BSides CT 2025

So we have our next talk for the track two is how EDRs see everything until they don't by Jacob Kat. >> All right. Uh hey folks thanks for checking out my talk. Uh a little about me. My name is Jacob Khallet. I graduated back in 2021 from University of Hartford uh computer engineering and I've been working as a sock analyst since. Um this is my first time giving a talk so uh bear with me here. Uh but this presentation is going to offer a quick but comprehensive look into the instrumentation points that Windows provides uh to endpoint detection and response products or EDR. We'll explore how EDR solutions collect that telemetry and provide it to sock

analysts uh and they rely on that daily and examine the impact of when EDR tools uh instrumentation points are tampered or disabled. So some of the key topics are going to be AMSI, user mode hooks, ETW, uh the role of kernel drivers in telemetry collection, and uh and WFP. And and don't worry if some of these acronyms don't mean anything yet. We'll get into that later. And hopefully by the end of the talk, they will mean something. Um, so I mentioned WFP where one of the things we're going to cover is how EDR tools implement uh important features like network containment um and how attackers may be able to bypass uh those mechanisms. So hopefully you'll gain a clear

understanding of the mechanisms that EDR uses um and some of the implications of their compromise. Um, and just a disclaimer before we start, everything we're going to talk about today is how EDR is implemented on Windows devices. Um, so just a quick overview, uh, we're going to cover, uh, get some background, get everybody up to speed with how EDR works and overview of some of those instrumentation points. Uh, then we'll dive into the ones that we're going to discuss. Uh, and then eventually we'll we'll wrap up. So uh if you don't know EDR stands for endpoint detection and response uh tools like Sentinel one, Crowd Strand, Microsoft Defender for Endpoint are the natural evolution of legacy antivirus.

Uh like legacy antivirus, they do provide signaturebased detection uh and file scanning. Uh and this is great for known threats. Uh but this doesn't cover you against anything novel. So uh where EDR comes out ahead a legacy AB is behavioral detections. So meaning the EDR tries to correlate several actions taken by a process and determine if those actions are something that malware would do. Um so the next question is how is the EDR tool informed of the process behavior and the actions taken on the system? uh the EDR tool will collect uh telemetry from many different instrumentation points um on Windows devices and and when I say instrumentation points I'm just referring to the method that's used to

collect telemetry. Um so the EDR tool that I used uh for this research had twoish components that acted as sensors on the device. um a user mode component uh which consists of a Windows service, an AMSI DLL, and a user mode hooking DL. Um and to my knowledge, that's all that uh there might be some smaller components in the user mode. Uh but the other component being the internal mode drivers. Um there's a handful of them that are installed by the EDR that serve different purposes. uh but both the user mode and the kernel mode are vital to uh maintaining a high level of visibility onto the system. Um so and by visibility I'm just I'm

just talking about the ability for the EDR's tool the EDR tool to send that data to the third component uh the cloud component. So that's where the EDR ships the telemetry off to for further analysis, correlation and ultimately review by uh sock analysts. So um the visibility of the EDR is only as good as its ability to get those logs to the cloud cloud component. So if you have no instrumentation points or you can't send that data to the cloud, then we have no telemetry and ultimately no detections. So uh moving on into our first instrumentation point here. Now that we have the baseline uh we're moving into user mode hooks. So user mode hooking is at a high level

a way for edr to gain visibility into uh Windows API functions that a process is calling. Uh they accomplish this by patching the address of a self of select Windows API functions in process memory. um so that when the function is called the execution takes a detour through an EDR DL. U so now let's dig a little deeper. Uh here's what I found while digging through a EDR's patent for this capability and how they accomplish this. Um, so when any user mode process loads a DL that the that the EDR wants to hook, the kernel mode component of the EDR is notified of that image load effect. The DL is being loaded. uh the kernel load component is notified and

then the kernel mode component will build an asynchronous procedure call or APC uh which is scheduled to the main thread of that process that's execute and is executed the next time uh it's scheduled. So that APC just loads the EDR's DLL component into the source process and the component will pass the address of the important functions that it's interested in which we'll get into in a minute and uh ultimately hooking them. So uh malware is executing it's calling that hooked function. uh when it's called the execution is passed through a redirect function in the EDR binary only after being called um in a trampoline function which is a kind of a funny name for it but the trampoline function is

just located in the slack space of the uh it's instructions that are located in the slack space of the hooked process and uh it just bounces the uh the the execution to the edrs uh DL's redirect function. So um they receive that info about the hooked function call in the EDR DL and uh it processes that info and passes it uh to the kernel mode component where the kernel mode component makes a determination on if that function call is suspicious in any way or if it needs to take any action. So if it doesn't need to block execution, it'll pass execution back uh up the stack to the to the original uh process and allow them

to call that function or they can uh take action against the process and not return execution back to it. So um uh so before diving into like a specific attack and what we might see um is there sorry it's a little small there uh but the uh we can use this tool called telemetry sourcer that's the logo up there that little guy. Um, and uh, we're we can use this to list out all of the DLS that are or all of the functions that are hooked by our EDRS DL. So, um, up on the on the left column there, you can see that all the functions that we're hooking are in NDLL and we have a list of, uh, pretty that

functions that are commonly abused by malware. Um so uh let's walk through an example of uh process hollowing and what the EDR might see when an attacker performs process hollowing. So um the malicious process starts and it tries to uh load NT DLL. The kernel mode components notified of that image load event and injects the EDR hooking DL into the process and the functions uh that we see here are hooked. Then the malware starts the process hollowing. So it'll create a new child process in the suspended state. Uh at this point the EDR would get a notification of the new process starting at the kernel level. Uh the malicious software would then call uh entap view

of section to unmap that process and uh essentially hollow out all of the original code from the the process that was starting. Um and then it might call allocate virtual memory to set a region to uh to write the malicious code to in that remote process and then write virtual memory to fill the malicious code into that region and finally to set your uh the context thread uh to get your start point and then uh resume thread to start the uh the execution of your malicious code in that remote process. So uh the important thing here is that the we can see that allocate virtual memory is hooked. We have uh set context thread get context thread on our view of

section all of our uh all of the functions that are used by malware to uh perform to perform process hollowing. So, uh, if the EDR is paying attention, it will note that these event these functions are being called in order and that that behavior exhibits process hollowing and if it's paying attention, it'll hopefully raise a detection. Um, so let's move into uh some evasion techniques now. So the going to go over two common evasion techniques for uh for bypassing user mode hooks. So we're going to uh one of the main ones is just manually reloading into DLL or remapping uh the DL. So because the DL the version of the DL that's in memory in the process the

functions are rewrit the function addresses are rewritten to uh send you to the E to send you on that detour to the EDR's DL. So what we can do is load up a new uh version of NDLL into memory and uh a clean version from disk and then write the contents of that clean NDLL uh text section into the hooked NDLS.ext section in memory overwriting the user mode hooks and then uh being able to directly call those functions. Um a more for full foolproof uh way would be to do direct SIS call invocation. Uh so if the functions are hooked you can uh just bypass the function and go straight to the sys call. Uh all those empty functions that

are hooked are essentially just wrappers for sis calls. So uh if we know the sys call number and the parameters we can just directly implement the sys call stuff without ever having to interact with ntd. Uh so that's user mode hooks. We're going to move into AMSI now or the anti-malware scan interface. So AMSI is an API developed by Microsoft that allows developers to essentially uh opt into sending uh sending content to endpoint security agents. Um you can see that uh you will see that MSI is commonly used by uh scripting engines like PowerShell or WScript uh apps that work with untrusted memory buffers uh and apps that interact with nonportable executable code like docx

files pdf files. Um so is already integrated into uh many common windows components that are abused powershell.net net, JavaScript interpreters, Visual Basic script interpreters, and the list goes on. Uh, this became a necessity back around 2015 when fileless and scriptbased malware became a lot more prevalent. Um, the debut of Empire uh kind of pushed uh pushed the the file and scriptbased malware into the mainstream and uh yeah, this Microsoft came came head on with this. So, how does how does it work? uh first time at the time of installation the EDR will register an inprocess com server with Windows. So first it sets up the com class under the registry tab. Uh the first one there setting the value of in

process server to be our EDR's AMC DL. And then the second one is registering that COM class with AMC. So it's saying uh it's it it registers the com class so the AMC knows that when we need to perform scanning we're going to use that EDR's DL. Um so if you have an EDR product I encourage you to take a look under there and poke around at their DL for AMY. Uh so the DL is registered and uh it's going to be used with AMZ but now how is it actually implemented by PowerShell or whatever whatever scripting engine that's being used. So the when PowerShell starts the system automation uh system management automation DL which just provides the

runtime environment for PowerShell uh will initialize AMY and check for any registered providers before executing any code and then uh by default power p PowerShell uses AMY to capture and scan script content using functions implemented in AMZY.d. So we'll see when execution of the PowerShell script starts that uh PowerShell will open an AM session using AM open session that's implemented in that AMD DLL and then we'll uh uh that will just be give AM the ability to correlate multiple scans. So we're executing uh multiple scripts we'll be able to see them in tandem. Uh then it invokes the AMZ scan buffer which is just uh passing the script content as a parameter and it will do

some input validation before calling the AMZ scan function which just iterates over the registered providers and then uh invokes the scan function that's exported by the AMZ providers DLL and at that point the EDR tool can inspect the script contents and make a determination and send that back to NMC where the determination can just be known good uh detected, not detected. Um so if you ever see like PowerShell scripts in your EDR telemetry, uh this is one place that they can come from. So, uh starting to talk a little bit about evasion. Now, um something interesting that an EDR tool might do is it might actually hook some of those AMC functions that are implemented in one of the AMC libraries.

Um, so, uh, when I did some digging on mine, I found that they hooked, uh, a function called, uh, the anti-utills function scan content, which, uh, actually this researcher Matt Grover in 2016 tweeted out this oneliner that uh, was used to bypass AMY and all that did was it just set this AMY init failed variable to true. And uh that ultimately just exited as soon as the uh AMZ in it and called this scan content function. Uh it would cause the uh it would cause the scan to fail before even starting because it thinks it's not initialized. So um what my edr did was it hooked this function and it said uh when that function was called it passed

the execution off to the edr dl because it was hooked and it said is emit failed true I don't care scan it anyway um and so that's just one way that edr kind of and attackers go headtohead and have to develop against each other uh as far as evasion techniques go. Uh, another one would be uh, patching AMZ DL functions still really common. Uh, AMZ open session and AMZ scan buffer are the two examples of the most commonly patched functions. Uh, one common way to bypass a receipt is to patch the AMZ scan buffer to uh, always return AMC result clean. Um, but because that's widely used, it's also widely signatured and it would require a pretty decent

amount of opuscation to bypass a modern PDR. A newer common bypass for AMY is the hardware break is using a hardware breakpoint uh on the AMZ scan buffer function paired with a vectorred exception handler to modify the parameters passed to the function. So or or to tamper with the result determined uh the determination returned by the function. So uh you can also tamper with the scan buffer length. So if you can manage to uh inject a scan buffer length of one uh it will always return a clean result because it's a one bite long script. Um or you could update the return value to zero indicating a clean result after the scan and just return execution back to

the caller. Um and now for a uh more interesting one. I actually have a little uh did a little work to to show how this how disabling ETW which is event tracing for Windows uh how disabling some providers you can uh see the direct impact on on EDR traffic. So uh reeling back now ETW is just a mechanism that Windows uses to log events that are raised uh by either a user mode application or a Chernobyl driver. There's three main components. Uh an EDW provider, a consumer, and a controller. The provider is just an application that emits events. So by default, Windows actually registers around,00 uh ETW providers and uh there's a few up here that EDR is a little bit more

interested in. Uh so we have you know uh being able to capture DNS traffic uh SMB client so your fileshare activity net runtime and WI activity um and moving into controllers now uh controllers are responsible for defining and ultimately controlling trace sessions in ETW which just record the events that the providers collect and then send them off to the event consumers. So uh if your EDR product has a controller with an open trace session, you can use log man uh to see what the session name is and then you can query on that session name to see what providers are enabled um in that session. Um and I also recommend checking out Flux Resourcer again. Uh it gives you a nice

list and an ability to disable a provider or stop a session. Um, so you can see how your telemetry is impacted. So, uh, now we know how ETW works, but one of the events that the EDR receives from ETW actually look like. So, uh, before we get into that, it's important to note that there's two ways to be able to read events from ETW as a consumer. It's either from a log file or in real time. uh nearly every edr tool is going to read them in real time, but for the sake of just showing this, I pulled up uh the log in event viewer to see what a uh a log from the WMI activity provider looks

like. So, um this I pulled this up after I ran the command wic process list. Um, and we can see that in the operation field, there's a query in there. Select star from wind32 process. Um, and so with this, the EDR will be able to gain some visibility into if there's sensitive WI queries running on the device. Um, so I did some testing to determine how removing the WI provider as well as the net runtime provider impacted the EDR's ability to detect misinumeration technique. Um, what I did was I set up a havoc beacon, the C2 framework havoc. I set up a beacon on my test machine, reflectively loaded uh sharp wi uh which is just a module to run WI

queries. Um, I reflectively loaded that into the beacon process and executed that same process ination query using Sharp Wii. and uh went while the provider was enabled. I got two events, one for the sharp wi.net module being reflectively loaded and uh as well as an event from a suspicious WI query being run. And then I reran the test after disabling the provider and the EDR lost its visibility at those events. So this is the extent of the testing I did with ETW. Uh there's a ton more uh providers to play with. So um and it's really easy with the tool uh telemetry sourcer. So if you uh have the opportunity to I encourage you all to go out and play

with it. Um so that's ETW and now we're going to we're going to get a little bit lower level into the kernel. Now um so EDR products may have many kernel mode drivers that serve different purposes. Um some may be responsible for providing detection and response capabilities. uh the threats that may have compromised the boot process. Um, additionally they can uh they can be used to implement device control. So blocking uh blocking USB mass storage devices. Um but what we're going to focus on is the ability to collect telemetry in the kernel. So uh the Windows kernel provides a ton of uh ton of telemetry to kernel drivers and they're going to accomplish that using the uh notification callback

routines and many filter drivers. We're going to start off with the notification callback routines. uh these are functions that are implemented in the EDR's kernel driver and are registered with the kernel to be invoked when a system event occurs. So a few example uh few examples of notification callback routines that can be registered are here. Um so like you said the EDR EDR's kernel driver will create and implement a function uh to be invoked whenever a process creation event occurs or a thread creation image load. Uh that's what we actually talked about earlier where those uh were those image load events and how they're used with the user mode hooking component. Um so that's what we're going to kind of zoom

in on next. Um so first the kernel driver registers the callback routine using the pset load image notify routine and passes in the the address of the function that will be invoked when that event occurs. Uh and the kernel driver also has to implement the the callback routine and with that it'll uh it'll be set up to receive an image name process ID and some info about the image including signature uh signing signing info as well as uh uh virtual address location image size stuff like that. Uh so uh after it receives the so say an image is loaded and uh after it receives that call back routine it's or the call back it's the EDR's prerogative to do what

they will with that info. So they can um like we said invoke the user remote hooking process or they can also use it to log your image load events to the telemet to your cloud component. Um so uh again if you run uh telemetry source for a system it will give you a great uh view of your kernel callbacks that are registered by uh by your edrs kernel driver. Um and it also gives you a great button to suppress them. Um so that's what I did on a device that had my EDR product. If I suppressed each call back one by one um and tested a default uh havoc beacon payload just default fresh out of the box um and I noted before I

suppressed them all the events that came in um and then retested with the callbacks disabled. And as you would expect when you disable the process create notification callback routine your process creation events aren't going to be there. Um but I was also surprised to see that the process creation events in the telemetry were also suppressed or also disappeared when I suppressed uh the image load and thread create. So there are some dependencies between the different callbacks. Another noteworthy operation uh observation sorry was the uh when the image load and process create notifications were or routines were suppressed no process was injected with the user mode uh hooking DL component. So, uh it does appear after doing all

this testing that the highest bang for your buck uh routine to disable would be the process create uh routines. Uh because not only did it just disable everything we just said, but it also uh suppressed the detection that was raised for that default beacon running um as well as any network events uh like DNS request logs and TCP connections which I thought was interesting. Um, I think it was because it didn't have a process to associate those events with, so it didn't know how to log them. Um, I did some additional testing with other callback routines, but I don't think we have time to dig into all the different use cases. Uh, so I just going to move

on. You could talk to me later if you want to know more about it. Um so uh moving into the file system uh minifilter drivers now uh or mini filters uh they're used by endpoint security products to modify file system event or to monitor file system events. Um so uh the image on the right there shows your uh mini filter architecture. Uh you have your IO manager as a main component filter manager your down at the bottom there you have your main uh your EDR's mini filter driver and on the right you have your file system driver. So uh when a request is made with a user to interact with file say for example creating a file uh the request is passed

through the IO manager where the request is handled and handed off to the filter manager where the filter manager will look for uh registered callback functions in each mini filter uh in order of al descending order of altitude. And altitude is just a way uh that it's just a number that's defined when the mini filter is registered that specifies how important it is that it sees the call back first. Um there's two big altitude ranges used by EDR products. Um here most common is the the antivirus range. And uh this is important because altitude is important to note because two s a system can't load two mini filters with the same altitude. Um so Microsoft actually

maintains a community list of uh of filter altitudes for uh any any minifilter provider. Um so like we just talked about the notification callback routines, mini filters also use callbacks. Uh when registering a mini filter call back, the driver needs to specify the major function and if the call back will be invoked before or after the operation is completed. So that's pre or post operation. Um and uh my er registers uh a lot of these major functions uh likely uses them to log file system name pipe and interprocess communications uh and to test what events are impacted if these callbacks are suppressed. I decided to use the lockbit 3.0 builder that was leaked back in 2022 to deploy

ransomware on my test machine. uh one with the call back suppressed and one with uh not suppressed and one with all of the callbacks suppressed to see what we missed. Uh as I as you would expect, all the file written delete events were uh nowhere to be seen. Uh detections for ransomware file access patterns were also nowhere to be seen. Um, but there were some interesting findings as well that likely uh had dependencies on these callbacks like a detection for a handle being open to the Elsas process. Um, an application event log being cleared and weirdly enough uh the process starting process start events and DL loads were also absent. So, um there's definitely a

dependency between uh all of these mini filter or mini filter and notification callback routines that if you pull one out, uh they all start to crumble. Um last thing I'll say about in the kernel here is that it's a bit more difficult to be able to suppress uh these than the previous instrumentation points because of the security boundary between the user mode and the kernel. So, uh you will need your code to be executing in kernel space. Um most common way to do that is uh to exploit a vulnerable driver uh that's already installed or by bringing your own. Um but vulnerable drivers are super common. Uh for example, millions of Dell Dell PCs were

shipped with a vulnerable driver DB util from about 2009 to 2001 uh 2021. And uh so if you have a if you have a older Dell PC, you probably have this driver on your device. But uh last bit about this is if you want to see how a vulnerable driver can actually be used to disable callbacks, check out this project edr sim blast. Um here to the last topic WFP we doing on time. Um so WFP is a platform uh that Windows provides that allows external software vendors to mod modify, monitor and filter TCP IP traffic. Um so I used the tool WFT Explorer to find uh the ADR I was researching with used this

functionality. uh it uses it to likely to log incoming in the cloud network connections, implement hostbased firewall capabilities and uh network contained devices. Uh so the the WFP architecture is a bit complicated. So we're just going to abstract a few of the components that are relevant today. Um a WFP provider is one of our main components. It's uh provides uh the provider is just an object that is used solely for for management purposes. It's upon creation of a filter, you can specify a key and that just associate associates the filter with in this case the EDR service. Um and then there's filters. Uh so WFP uses the WFP filtering engine which consists of a user node component, parental node

component and together they perform the network filtering operations that u map loosely to the operating systems uh networking step players. So uh yeah, you can configure that uh the the filtering by just uh in a user mode application or in a kernel mode application by just adding a WP filter to the WP filtering engine. Um when creating a filter, you just specify a weight. So how important is it to uh to take that action uh specify an action and uh a condition in which to take that action. So, uh, we're just going to talk about the permit and block actions today, but there's a couple more, um, but we won't get into those. Uh, so this is an example of a WP filter

created by my EDR product. Uh, the this is the, uh, conditions tab. The other tab will show us that it was created with the action permit. Um, and you see that the condition specifies the local port hex 7440, which is 29760 decimal. um in the protocol TCP. So I actually just ran that stat and found that there was a connection with the local port 29760 uh coming from paid 4 which is the system process and uh the remote IP appeared to be a IP address of my uh EDR products cloud component. So, um, this was just an example of the EDR using a filter to ensure that another filter won't won't uh compete with the

EDR's ability to to send telemetry to the cloud component. Um, so when a device is network contained, you'll see uh that the EDR product uses several very broad blocking filters. So uh and those will be paired with permit filters to allow specific EDR IP addresses as well as allow some some specific services like DHCP. Uh so this person Neero 1010 uh actually wrote a tool two years ago called EDR silencer that uses WP uh to block network connections from user mode uh processes for EDR products. uh essentially just blocking all of the telemetry coming from the ADR. But this didn't work on my EDR because the telemetry isn't sent to the cloud component from a user mode process. It's

sent from the uh the kernel driver. So a colleague and I ended up researching uh WFP and writing our own tool that uses the WFP API to uh block EDR telemetry uh by adding WFP block filters on dynamically resolved EDR domains. So um that along with two other features. The first being to remove uh blocking filters by that are put in place by the EDR to essentially lift the containment uh if the device is network contained. And uh a third case where we essentially network contain the device uh blocking all traffic except for traffic to our command and control IPs. Um, so, uh, we aren't going to go into this too much for the sake of time, but,

uh, here's a quick look at how we were able to lift containment. And, uh, if you want more detail, you can check out my blog post, uh, linked in this slide, which I'll share up later. Um, so this is a screenshot of WP Explorer showing the block filters put in place by the VR product. uh when the device is contained. If we looked at the conditions, we'd see that they are applying those broad blocks across TCP, UDP, connect, receive, accept attempts uh for IPv4 and IPv6. So uh and not pictured here are the the uh filters that allow the telemetry, but I just did a simple test to make sure the device was network contained. Also

confirmed in the EDR portal and uh the curl command that we tried did not work and was not able to reach out to any network. And then we uh ran this tool that we wrote called WFP evasion that uh with the command uh lift containment. And this just loops over the block filters put in place by the EDR's WFP provider and attempts to delete them. In my case, mine didn't my EDR didn't have these filters secured in any way. So with administrator privileges, I was able to uh just loop through them all and delete them with the basic uh WFP API provided by Windows. Um the ter terminal on the right just shows that uh

after after deleting the filters, we were able to uh get out to the internet again and uh there weren't any detections for this. There wasn't any telemetry. Uh additionally, the device containment status remained contained in the EDR portal. uh but uh the changes were worked indefinitely and persisted through a reboot. So we ended up uh reporting this to the EDR vendor and they fixed the security of WP filters recently. I know who uh but the uh the no longer allowing the removal or modification of their WP filters and they actually added an event to their telemetry for suspected WFP tampering. Uh, I only tested this on one EDR product though, so if you have a

different EDR product, I encourage you to test it out on yours. Uh, it's pretty simple to replicate. You just, uh, use the WP API. It's pretty well documented and just go ahead and remove your EDR's uh, containment filters. Um, okay. So, that's uh, just about it for content we have today. Uh but today we covered a bunch of instrumentation points that EDR uses to collect telemetry from Windows. Um some that Windows provides and like user mode hooks uh some that the EDR has to make for themselves. Um as we showed there's plenty of ways to evade and tamper with EDR instrumentation capabilities. Um and uh as I just demonstrated the EDR didn't secure the WP filters. So disabling critical

capabilities like network containment that are important during uh incidents is uh it's was as easy as deleting a filter uh with administrative privilege. So hopefully this talk inspired you to question your EDR's ability to protect itself and do your own research and verification of what they market to you. Um future research opportunities. Microsoft announced in May that they're working with DDR vendors to provide the same level of telemetry that they're currently providing, but to move sensors out of the kernel. Um, so I'm excited to see what comes there because there'll be new instrumentation points and ultimately new evasion techniques that we can play with. Um, and I definitely recommend you check out Matt Han's book,

uh, Evading EDR, if this content was interesting to you. I referenced it a lot when putting this content together. So, um, but yeah, that's all I got. Thanks everyone.