Hunting kernel rootkits with eBPF by Asaf Eitani & Itamar Maouda Kochavi

Hello everybody we're very excited to be here and present our research about a kernel rootkits and how to detect them using eBPF technology. In our research we analyzed the most common eBPF kernel rootkits and found common behavioral patterns between most of them. Colonel root kids has always been one of the most sophisticated tools in the in the APT toolset, but nowadays we see more and more emerging attacks that make use of this advanced malware. We will take you through a journey into the deeper workings of Kenner rootkits, how they operate and how to detect them using our open source project using eBPF. Along the way we will explain about eBPF technology and why it's so great for

our cause. But first let's say take a quick look about ourselves. I'm a survey Tony said from team now the loose our security research team aqua is a security company that Does cloud native application security and that's the reason that we wanted to? Do Linux security so much because most of the clouds cloud is operating on Linux in my past I've done mostly Incident response and other forensics areas and nowadays I'm working mainly on eBPF and low-level Linux research and I'm a Tamar Muda I'm also security researcher at a team now to lose and My previous experience is in Malu research and Linux internals. And more than that, in the past six months, I mostly focused on research what gets behaviors in

the Linux kernel and also to create the detections with eBPF. And actually, this is me and the staff's first time at a business as speakers, so we are very happy to be here. Okay, so what will we have today first of all we'll take a quick scope about the kernel what it is and then we'll proceed into seeing what tracing methods are available to us in Linux kernel and We'll then elaborate on eBPF and how it works and we'll move on into a quick scope about rootkits and what they are. Then we'll present the result of our research which is how to detect Linux kernel rootkits and how they operate and at the end we'll see what's

next for our open source project Tracy. So I'm sure most of you already know what the kernel is, but let's just do a quick scope. So the kernel, every host or computer is divided into the user space and the kernel space memory. The user space is where all the user space application resides, such as email programs, text editors, or internet browsers like Chrome or Internet Explorer. May you rest in peace. We will not be seen. Kernel space is where all the operation system resides. So all the file management privileges and stuff like that sits in the kernel module, in the kernel space. And we have syscalls, which are the intermediate between those two spaces. So there's sort of like an

API between the user space and the kernel space. In this talk, we'll talk mostly about kernel rootkits, but there are also user space rootkits. So when we're talking about detection, we have to talk about tracing. Tracing is the act of monitoring actions performed on a computer to be able to see what was happening. So in Linux kernel we have a lot of options but we want to show, we want to choose the one that is right for us. So first of all we need to make sure that our tracing method has a low overhead so it's efficient as possible in order to keep our other programs working. Secondly, we need to be stealthy and have a strong security boundary because we don't want the attackers to be able

to know that we are tracking them or disrupt our actions. Next we wanted to have full system visibility in order to see as much as we can obviously. And last but not least we want it to be safe so we won't crash our system or even um disrupt our other user space uh or kernel space uh programs that are running correctly. So in front of you is the list of the kernel Linux uh uh of the Linux tracing methods. The first two are Ptrace and library injection which are user space tracing methods. As you can see, they don't quite fit our needs because either they have high overhead or they're not as stealthy and as low security boundary. Both of which also doesn't have a

full security view because they obviously reside in the user space so they can't see anything in the kernel. Then we have kernel model which was the go-to solution into a tracing method in the last couple of years which has almost all of our required features but the problem with the kernel model is that it is not safe. So the problem is if we have a bug or an error in our kernel model we can cause a crash of the system or even create a vulnerability for attackers to exploit. And then we get to eBPF. eBPF is sort of a new technology that allows us to have almost all the advantages of Linux kernel model, but have it secure because eBPF is

essentially code that runs in a sandbox inside the kernel. It is also supervised, so the meaning is that the code is verified before it's compiled and it has checks to check if you're accessing pointers before checking them so you won't have null pointer reference and stuff like that. eBPF also has the ability to use a kernel feature that's called probes. It allows us to place hooks on our computer and kernel functions. So what is hooking? If we have our normal execution flow, hooking is essentially the act of attaching oneself between two functions and getting the arguments of the called functions. That's a That allows us to run our own code and also get the parameters that were transferred. So what are the types of eBPF probes that

are available to us? There are many types, we will focus on three today. The first one is kprobe. kprobe is a dynamic kernel hook, which means that we can almost attach ourselves to any kernel function inside the kernel and run our own code. We also added LSM hooks which are places within the code that has security meaning to it and are meant to be attached by a cable we will elaborate on that in a second trace points are Much like a k-probes, but they are static. So they are defined in the kernels Code and we can use them, but we can't just decide where to put them. So And uProbe are again really similar because they are dynamic user hooks which are attached to the user space

programs rather than kernel space. Okay, so why did I mention LSM hooks? We'll demonstrate the problem that we're trying to resolve to avoid with LSM hooks that named TOCTU. TOCTU is time of check to time of use and it's a race condition vulnerability. This vulnerability is why the attacker can affect what we're seeing inside our tracing method. We'll give a short example of how it works in a process creation in the user space. So let's say we have the attacker and he has a thread. He wants to spawn a new process named malware. So he writes the path to the memory and gives that pointer to the execvc call which is how you create new processes in the system. memory and write it to a file

struct in the kernel and to be saved for later usage by other uh functions. Now at this point uh the threat actor has control over our user space uh pointer and we can also it can also uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh

uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh

uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh

uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh uh Use race condition in order to change it before we even monitor it So you will create another thread and write over the same path with a legitimate path like being a less then our Cisco trace point which is a trace point which is vulnerable to talk to it will read our the malformed Path and save it for later use to report to our tracing method. So As we can see, the execution continues deeper and deeper

into the kernel, into security BPR uncheck and sketch process exec, both of which has an LSM hooks and kernel trace points we can use to be a better solution and not be vulnerable to talk to. The reason is that because they're reading the kernel pointer rather than the user space pointer. Okay, so LSM will help us avoid talk to but they also have some neat perks that we'll definitely take advantage of. The most significant ones are a relative and symbolic links resolution. So if an attacker writes a relative path, we'll get the absolute path to the actual file that will be executed upon the final function. So now that we've seen eBPF and why it's so great and different from

other tracing techniques, I want to tell you a little bit about our open source project. Tracy has been built to take advantage of eBPF advantages and to be able to see more into the system. So we have increased observability into the kernel. We also have some neat features that are like artifact capturing, which means you can dump pickups, memory regions, files that have been executed and save them for later usage and we'll see a demo on that later. And we also have a behavioral detection, which means that we have a rule engine that we can write rules on the events that come from Tracy. This project is open source and available in GitHub. So before we proceed to our

next topic which is rootkits, we want to show you a quick video from our favorite movie Mission Impossible to help demonstrate the idea. As you can see we have our hackers in official access. And they're using technological means to fool the security measures.

Now as we've seen, this is a metaphor for our current day security measure, our old school antivirus sees nothing but in reality there's a rootkit that's hiding a malware behind it. So what is a rootkit? A rootkit is basically a malware that it purposes to hide artifact processes and stuff like that in the system. So it does that by hiding processes, files, network connection, kernel modules and even itself. That obviously makes it really hard to detect because most time you won't even know that it's there. Okay, so now we're very excited to show you the actual result of our research which is the kernel rootkit attack flow. After we analyzed for six months every rootkit that

we can place our hands on, we came up with the common behavioral which is the following. Voila! We'll take you step by step to each stage of the kernel module and see the kernel rootkit and we'll see an example for each of them. So the first thing that a kernel rootkit would want to do is it want to be loaded into the kernel. So of course we have kernel module loading. The second stage after it was loaded it needs to hide itself in order to remain hidden. So that will be the second stage. Next it will place hooks upon the system to be able to hide other artifacts like processes and files as we've seen. and it will proceed into establishing a communication channel to be able

to receive commands from its operators, either C2 servers or other user space malware, which will be the intermediate between the operator and the kernel module. The last stage, of course, is to execute said command and just give the operators to do what they want. It's important to say that once the kernel module is loaded, eBPF and kernel modules are operating in the kernel, so each of them can disrupt each other and whoever gets the upper end will win. So kernel module loading, there are two syscalls that can be used in order to load kernel modules. The first one is finit module and second one is init module. They're very much alike but finit module loads a kernel module from a file and init module

loads a kernel module from a buffer. You can see that they both have in common the call for do init module which is how we're detecting a kernel module loading. Okay, so next we'll see a demo of how we're capturing and seeing that a kernel module was loaded, but we also see a capture of a fileless kernel module. That's really important because if a MARU runs only on the memory, we won't be able to investigate it and take it for reverse engineering or analyzing. So as you can see we have our uh Tracy command above with all the relevant events and magic write uh which indicates a file being dropped uh you will see a file being dropped to memfd uh memfd

is a memory uh backed uh directory which means it's the memory it's not really the disk uh and also we've supplied with the dash c module uh meaning capturing modules. So in the bottom process part of the screen you can see that we have a vulnerable NGINX uh that's running in the background and uh we're running Tracy to be able to detect any uh actions that were performed. Then we'll run our exploit which will cause three events to pop up. Magic write with the path name memfd not malware and we also get the first bytes of that uh file so we will be able to determine if it's a binary or a text file or whatever. And then we get two events, the do init module and f init

module, which indicates kernel module loading and also we get the name of the kernel module, the version, stuff like that. At the bottom part of the screen you can see that we were able to capture said kernel module and take it for later investigation. Okay, so now we loaded a kernel module. How do we hide it? So before we try to hide it, we need to see how kernel module seems in the memory in Linux. Kernel models are stored in a linked list in the memory and the way that they operate is there's a pointer to the first one in the system and then in order to enumerate all of the kernel models that are currently loaded, we're just going with the next pointer until we

reach to the end and complete the loop and then the program knows that we finished to iterate over all the kernel model. What a kernel module or a kernel rootkit could do is modify this uh those pointer the next and previous pointer and actually detach itself from the linked list. By doing that the um the iterating program will just skip over this particular kernel module uh it will still run in the background but uh you wouldn't able to see it. Iter martiu. Thanks Asaf. And following what Asaf said, the attacker loaded kernel model and hide it. And this is the time to do some fun. And by fun, I mean to start hook functions. So we mostly focus about two out of these three, because the

last one is pretty much similar to the second one. And the first one is the system called table hooking. This is a technique that targets specific entries inside the system called table. The second one, which is more internal, is the file operations hooking. And this is a unique technique that overwrites specific operations handles of a specific file in the kernel. I know it sounds a bit confusing, but I promise we will get to it in a few minutes. And the last one, which is the sequence operations, it's pretty much the same technique as the file operations, but it works on a different struct in the kernel. But for now, let's start with a system called

table hooking. So at this point, we know what is syscalls and what they're used for, but who's actually managing them? How are they stored in the kernel? So basically, they stored in array where in this array, each index representing the type of syscall. And in this index, it contains the address of the function that will implement the syscall. So if we'll take a look over this diagram, which illustrates a scenario that we try to list a directory. So we will call to the ls command, and the ls command eventually will call to the getdance syscall, which this syscall is used to get the entries of a given directory. So what eventually will happen is that the application in the user mode will call to

getdance, then to actually execute the kernel will go to the system call table to the entry of the a Giddens, which is a 217 and then we'll execute the function there and as you can see a guidance will of course a parse the directory and bubble up to the user application the result and as you can see in the in the little photo under that is that we've got the malware file and So, commonly, because an attacker can load a kernel model and override entries in the memory, He can just go to the system call table and override the address that's stored there. And by that he can insert a hook between the actual call to the original syscall and to the executing from the

system call table. So what will eventually happen in this scenario is that we will run LS and the kernel will go to the system call table and execute the function there. The actual address will be the rootkit hook. So he will gain the control and most of the time he will call to get ends to get the data that he should return to the user. But at this point he will filter out the data that he may want to hide. And as you can see in our case, he will filter out the malware file. And in the result, the LS Qand will show us that this directory is pretty much empty. So To detect this

type of hook, we need to take a look at what happened in the memory. What is the memory layout in the kernel when we've got no hooks and what happened after we load the amorphine and deploy those hooks. So we've got two major boxes. The first one is the system call table. And as you can see, each entry points to the blue zone, which is called the kernel memory map. This is representing the code of the kernel compiled with the kernel. And this memory zone is bounded by two symbols, which is SText and DText. And we've got another zone, which is the red one, and this is the kernel module memory map. This zone is the memory zone that for allocations of a new kernel model. So

if I will load now a kernel model, it map to the red zone. And following to that, if I now insert a rootkit, you can see this happening. Basically, the rootkit will override the address in the system call table, and because the address of the actual functions of the function won't be in the kernel memory map because it's from a new kernel model. The address will point to the kernel model memory map. And by that, if we trace what we've done, we load the trace and then fetch those addresses from the system call table and compare them to a S-text and D-text in order to check if they are in the memory map region. So just before I'll show you a demo

of how we capture it, let's take a use case from the wild. Let me introduce you to the Amorphin. The Amorphin is an open source rootkit project that works on several architectures and versions of Linux. Basically, all of its hooks are based on system call table, and on system call table hooking. And he's got one thing that is very special. And we also saw it in several types of rootkits. And this is the that there is one type of hook that can hide two artifacts. And this is the hook on get_dents. And as we understand, get_dents is used to list files, and by that we can hide files if we hook this syscall. But in Linux, everything is a file. And specifically, processes are also files. If you take

a look over the Y diagram, which illustrates the structure of the /proc directory, we can see that Each directory, each sub-directory is with a number which representing a type of process. And in this directory, there is the actual data about the file. So eventually what happened, is that when we try to list files in Inox, we're just iterating /proc and then iterating over each directory and print the data about the file. So by that, if a rootkit can hook gedants, he can hide these files at these directories, and by that, when ph, for example, tries to list all the processes and iterate over /proc, it will miss those entries and will hide this process eventually. So

just before I'll demonstrate the demo, we will run Tracy several times. The first one is just to see that there isn't any hook on the system. Tracy will alert us that everything is fine. And then we will insert a diamorphin, which is the root key that we saw. And the diamorphin is loaded hidden. So as you can see, Tracey alerted us that it found three hooks over GEDN, GEDN64, and KILL. And it also alerted us that the hooking module is hidden. And this is because the diamorphin is hidden. But then we will unhide the amorphine and run Tracy again. And at this point Tracy knows to detect using the curl sims file which is a special file in the slash proc that the hooking

model is the amorphine. But can we go deeper? Maybe there is another abstraction under get_dents that an attacker can hook in order to hide those same artifacts. Well, basically the answer is yes, and one example of it can be the file operation struct in the kernel. So, as we said, everything is a file in Linux. And in the kernel, there is a special struct which is called the file operations struct. And this struct contains all the functions that will implement a given action on a file. So, if I will take, for example, the listing directory example that we've seen in the previous hook, we will execute LS. LS will call to this to get ends from the system call table but then what will actually happen

is that get ends will go to the struct file operation to the member of iterate or iterate shared and then execute the function there and this is the function that actually will iterate over the data and save the data about each file and because an attacker can load the kernel module and modify any memory. In the same way, it can just insert its hook and filter out the data that it may want to hide in the same way that it did in the syscall, just in a lower level in the kernel. And this is also cool for us because the same detection that we did in the syscall table hooking can work on that in the same way. So we'll take a look over this demo, which is representing

First, it will be representing a use case of process hiding. We'll first see that there is the hide me process, and JCC will alert us if there is any hook. And then we will insert phide, which is a POC that implements this type of hook. And then we will run ps again. JCC will alert us that it found two hooks. One of them is over the file operations track, and then another hook over the iterator specifically. And also it will alert us that the hooking module is p-hide in the same way by parsing the curl-sims file. And all of this is very cool and most of the times an attacker will gain control over some

artifacts in the kernel and compromise some of the system. But what will happen if, for example, an attacker will want to hide another process or file or change a flow of hook that he deployed? So for this sort of actions, we've got the communication channels. Communication channels are basically the way that can attack and control these mechanics in the kernel in order to update their behaviors. So this is a bit a war of creativity because in the end, it will be implemented in several ways. from a network packet to creating and overwriting special devices and files in the kernel, and also to override syscalls. This is just a work of creativity, and it's just an endless world. And to the final stage, Danny, the stage is yours.

DANNY SULLIVAN: Thank you. Okay, so we know that you are all probably very hungry at this point. It won't take long. So we've seen two examples of how we can place hooks on the system to be able to hide artifacts, the syscall table and the file operation, which operates both within an example of the get dense syscall. And we've seen how we can establish a communication channel. Next, we'll see an example for a command being executed. So the most requested command by attackers is process creation, of course. Now process creations, we got to the Holy Grail of the attacker, the command execution of course. Process creation, as we said in our talk to slide, is done by the execvc call, which is call in term, security BPRM

check, and sketch process exec. But what is interesting is that we have another function within the kernel that enables us to create a process in the user space, which is call user mode helper. Why is that interesting? Because most security measures and tracing methods will hook themselves upon the syscall trace point of execve to identify process execution. And by using call user mode helper, we can actually bypass all those measures and tracings because we call the kernel functions directly, the security BPRM check and then sketch process exec. Okay, so we see a quick demo of that. We run a normal command, cp, and as you can see, we have three events that are popped up, and then we

are loading our kernel module, which creates a process in the user space. So in the first box we have execve, security BPRM check and sketch process execs under the event. And as you can see in the second box we have instead of execve we have call user mode helper. So we have another neat feature in Tracy that to indicate that a process were invoked from the kernel and which is the flag on the bottom part the left bottom part of each square. As you can see, invoke from kernel, the first one is zero, the second one is one because we know it originated from the kernel. So what did we have? We were able to load the file as kernel module and capture it, detect that it was

loaded. We've seen how kernel modules are able to detach themselves from the linked list and remain hidden. We've seen how we can place hooks like syscall tables and file operations, but also sequence operation works pretty much the same to file operations. We've talked about how communication channels are really a world of creativity for the attacker and they can just use the imagination to create one. And we've seen an example for a command execution in this type of process creation and how we can bypass security measures. So again, everything that you saw here in our demos and much, much more is available on our open source in Tracy. You are more than welcome to check it

out. Thank you What we will have there next in Tracy we're planning on researching about other hooking a methods We're planning to incorporate use new probes to be able to hook ourselves on user space program as well And we're planning to incorporate LSM BPF which will help us prevent actions being performed in the kernel and not only detect I And as a parting note, I'm inviting you all to be a part of our community and contribute to Tracy and our effort against Colonel Rootkits. Thank you.