Scaling Runtime Application Security: How eBPF Is Solving Decade-Long Challenges

our first two speakers are ready to go gentlemen all yours okay so hello everyone H thank you all for having us here and thank you for coming to our talk uh it's a real honor to uh start the event um so uh today we're going to talk about uh scaling runtime application security and how ebpf is solving a decade long challenge but first let us introduce ourselves sorry so my name is Guy Kaplan with me the AI Lumi let have introduce himself yes so I thank you guys for being here so early my name is a lelki I have 10 years of experience in R&D I focus mainly on AI engineering and security research and I also really love to climb

mountains so if you're into hiking or mountaineering we can climb a mountain together and with me is Guy okay so I'm guy Kaplan I'm a vulnerability research lead at the CTO office of poo security I have more than a decade of experience with vulnerability research and software development and in my free time I'm also a scuba instructor uh so let's dive in so a bit about what we're going to cover in today so we're going to talk about rasp or random application self protect setion we're going to cover what it is how it works what people expected it to be and why we think it failed we'll later show you uh what is ebpf and

how we think ebpf is Reviving The Field of runtime application security H it's important to say we will have time for questions in the end so if you have uh questions please write them down and we will get to them later so the need to protect applications is not new and thanks for jgpt for this amazing amazing image so for centuries or decades maybe developers and Security Experts have sought ways to Shield software against against threats traditionally application security have been implemented by focusing mainly on the left side of the sdlc that is using static tools such as SCA SAS code reviews and so on to improve the security posture of the application while other realtime tools uh such as a

firewall or API protection Tools H do offer some protection in runtime they usually El The crucial application context that is needed to protect against many threats and recent events such as log for Shell spring for Shell H the exit back door and in general the vast amount of organization that are facing attacks uh proves that traditional security tools are simply not enough to block or detect application threats you ask yourself why so it's because these attacks they happen only in the run time and they require proper application context to detect and therefore usually not detected by existing tools so to address these issues uh a new category of security tools have emerged uh and was quickly adopted by

cisos all over the world so it was first introduced by Gartner in 2012 uh rasp which stands for uh runtime application self- protection is a Rand time application security solution that uses application Level software instrumentation methods to W the application's code in in order to enhance its uh Security in production rasp was the first solution that emerged in the market that attempted to provide a solution for application attacks in production and spoiler it's working but it's far from perfect but before we can dive into why Let's uh let us introduce Ras what it is how it works and why we think it failed so we already know what rasp is trying to solve uh but how does it do it

so rasp tools work by adding runtime checks and validations to your application in strategic places uh it operates by utilizing strategies such as software instrumentation um addition of faction decorators or adding dependencies during the build time to inject uh security tests into your functions during execution rasp evaluates inputs and outputs of the application in the applications uh context and Tres to determine whether an attack is taking place so with some help from jgpt I want to show you an example of how a rasp tool rasp rule might look like for detecting SQL injections I hope you can see something um but H what you see here in this image is some code that receives an SQL query as an input and it's try to

tries to identify whether an SQL injection might take place so it is looking for uh some SQL specific keywords um and the these keywords are usually something that attackers will try to insert when trying to uh assess the exploitability of SQL of SQL injections so the rasp agent later hooks into the SQL library that you use in order to query your database and it inserts these test before running the actual query so if if it sees something that is odd in in this context you can just block or at least detect these attacks um the RAS pagent can later H be injected into running processes uh this is what what is called instrumentation so this is an example of how you inject

using Java agent you can inject into Java code and you you can use LD preload to uh to inject code into your go application and but not that not all languages support this way of injecting code into them uh which adds a bit more uh complexity into deploying uh such Solutions uh so since jgpt tends to lie I've also included a real example too this is taken from open rasp by Buu but as you can see from the function name do real check without request real code tends to get a really messy so we will stick with the chpt from from now on okay so another way of implemen is Ras rasp is by adding uh these function

decorators so this is a Java function that is doing pretty much the same thing as we saw in the last example um it is defining a decorator and then to use it in your code instead of injecting the rasp agent into your code um you just need to wrap all the query query method that you use to query the database with this detect SQL injection decorator and then it will also run the the checks before running the actual query and you can decide to block them or do whatever whatever you like so in general rasp is meant to protect the application and detect the exploitation of vulnerabilities it is not trying to detect attackers that are

already inside your production environment it is usually this is usually handled by workload protection tools uh that focus on the on the host and are also typically used in in container security um and also note that each framework as we just seen have to have its own rules in order to have full coverage so while rasp can be efficient in detecting some application based attacks uh It suffers from some major pitfalls um let's let's H look uh let's talk about why we think it failed so as a security solution designed and implemented in the past decade rasp suffers from some major pitfalls which have led Security Experts to consider it is as an ed added layer of of protection

rather than a must have security solution friction unpredictable impacts on performance and stability and the deployment challenges is are just a few of its problems but the main issue is it doesn't scale let's see why so first and most importantly stability integrating into applications level code is a non-trivial and expensive undertaking that can potentially lead to stability and performance issues this is because due to the way Ras instruments your application if your rasp tool happens to have a bug and crash your whole application crashes and believe it or not bugs exist and application do crash even security Tools H and of course you cannot take a bet on production server stability any resulting downtime can have an impact on

slas causing revenue losses and damaging the companies uh reputation and Market position the second reason is the added friction so the GF here is actually how you how you add the rasp agent into your build build file um so as we've seen many rasp tools require the developer interaction in order to integrate with them when the cold or build changes are needed and the thing is developers are usually not measured for the security of their applications uh sadly but that's what abss are for developers are being measured for delivering features as quick as possible they simply lack the time and motivation to invest in the cumbersome process of uh integrating security tools which can be very

frustrating like why integrate a tool that only makes my product break and in addition the rasp approach covers only specific functions that are considered fragile or vulnerable in most implementations rasp offers security solely for the code that has been specifically wrapped by the user as we've seen with the decorators example thus leaving third party components with no protection at all and last but not least deploying Gras Tools in production is operationally challenging it's essential to acknowledge that due to the methods rasp is implemented with rasp requires distinct deployment and customization for each individual application and also when new threats emerge updating grass tools can be very challenging uh due to the complexity of the implementation and

deployment H which add another layer of risk of instability H in other words it's very difficult to re instrument and application when you update the rasp Tool uh without crashing or restarting it and indeed and indeed as you can see from this Gartner hype cycle report for application Security in 2019 rasp continued to prosper and almost reach maturity it's a marked with red but then in the report from last year it suddenly disappeared so it's kind of small but trust me it's not there H and just to check raise your hand if you have a Ras tool running in your production environment right now no one but the fact runtime self- protection is dead doesn't mean that

there is no need for runtime protection at all the market still Demands a scalable runtime security product we just need to find a better technology this technology needs to be seamless and frictionless ensuring that it does not disrupt the performance of the application and does not introduce developer friction it should provide comprehensive attack coverage monitoring the entire host rather than just the application it should be easy to deploy and maintain but most importantly it should be stable and we believe that ebpf can answer uh those needs so let me hand the speak to AI to present how ebpf is answering these challenges AI thank you guy thank you for being so awesome now let's let's talk about the role of BP F

uh in today's upsc EF is much more than just a buzz word let's begin with a high level overview of ef's strength I know some of you might be familiar with but I'll go go go over them quickly um ebpf is known as extended berly packet filter uh is a groundbreaking technology uh in the Linux kernel that allows us to run user space programs in the kernel in a very very safe way BPF programs are actually compiled uh to BPF code which you might be familiar with if you ever used IP tables for example so you've used the BPF virtual machine and just like IP tables it does not require reboot to apply changes to your program you can uh

do it without restarting the server unlike kernel modules which is a big difference and most importantly it is safe by Design and let's understand what it means the the ebpf subsystem uses something called the ebpf verifier it basically asss that our code is safe from infinite Loops uh invalid memory access uh and U that it basically does not hang or block and might end up crushing an application so it basically asserts that it won't crush the system why should I use BPF besides so it's very very Dynamic all you need is Linux you know maybe seven or eight years ago you wouldn't have said that but today uh it's not the case ebpf is almost on any Linux out there so you can

just pop SSH to Any Given machine hook a specific place inside the operating system and gain visibility like a zoom in uh where you need it the most without any further installation without risking your application ending up crashing and your application most of all can be already running you don't need to restart it in any way and ebpf is very efficient besides that uh thanks to the adjusting time compiler and the lack of contact switches uh ebpf results in very very low latency and we all love that but evf is actually battle tested uh in production by large companies uh such as Google Facebook uh Facebook is actually meta have been uh one of the early

adapters of ebpf and they've been writing BPF program since 2017 and Netflix who uses it for performance engineering uh you might know Brandon Greg which I'll mention also later uh he did amazing stuff with BPF at Netflix and the list goes on believe me EF robes allows us to to support many different kinds of applications so evf is very uh versatile and if you don't believe me there are 120,000 hooks that you can hook inside the kernel and gain visibility just like that through the BPF program 12,000 that's an amazing number that's basically every function in the Linux kernel can be hooked uh you can access system calls iio events uh scheduling events TCP UDP networking

events and uh even Conex which es and much much more and until now it was used only for observability and uh performance engineering um sometimes for networking but not really for security so I'm going to focus on a security tool soon and uh first let's understand what an evf program can access so unlike before uh ebpf allows us to hook all of these uh PS that I've shown before uh the most interesting ones are Trace points which are lightning facts uh they are integrated inside the Linux kernel already and uh these are just designated hooks that can be probed by the BPF program uh as as I said are considered the fastest among all of the probes and get this attackers

cannot avoid doing system calls this is why uh many security tools look at the system calls that your application is doing and remember that not all system calls were born equal so it basically tells us what application is doing and just like that and it's already being done you can uh look at the Cisco enter or exit event of any CIS in the Linux kernel and besides CIS there's also usdt usdt allows us to insert designated hooks inside the uh program that we run let's say if I have a python interpreter so it allows me to um insert specific hooks inside uh the the python interpreter such as function entry which you can see in here it's taken from an

open source that I've written uh that we'll see soon and um these are two first uh kinds of probes the first program will simply kill a python process if it uses os. system command it's just like three or four lines of code which I think is amazing and uh there are much more than just these two examples so it is used for networking purposes um actually Cloud flare is one of the pioneers and I've used XDP Express data path to mitigate and draw buckets early before they propagate to the Linux Network stack and reach IP tables which consumes much more CPU then dropping them early uh even before they reach the the network stack of Linux it allows us

to scale maybe from 1 million packets a second to five or 10 m packets a second which is amazing um remember that ebpf is only the building block and it takes much more than just these hooks uh to create a program that's actually helpful and there are some barriers to entry because EPF focuses solely on the interfaces with the operating system and if you use it right that it can allow us to do what's considered previously impossible um without without impacting the application's Integrity in anyway um but as I said there are barriers to entry you need to be proficient in Linux and understand ebpf ecosystem and it does not take just one day uh it's really hard to create code

that does not slow down the application so uh writing production grade logic is not um easy outside of a box and you need to know what you're doing and the ecosystem is limited so um it was mostly used for performance as I said and if we look at estr for example which is by the way is not suitable for for production environments it has an open Bug for at least 10 years in the man page of estrace that simply says a trace process runs slowly so you cannot do it and apply it on a production environment at least I wouldn't have and it leads me to the um to this point Performance Tools are not security tools if you know

Brandon Greg um um which is one of our gurus in vpf uh it's a famous say that I really love and just to prove it um I don't know this is the proof right estr is not suitable for production and the tools that we use today are mostly built around visibility and not around security so what is the connection between evf and security and who does it already so it is a trend we see large companies and small startups Implement security products that are based on ebpf if we look at Falco by C dig um there's a tetragon by celium and there's Tracy by Aqua security and it's just a few there is a trend and there's a trend for a

reason um it enables us also to do things like KSI uh which integrates LSM Linux security models with ebpf for enhan security hooks uh and we can actually control and block attacks if we detect them using evf hooks in real time and also there's memory access uh which is very important as you can see the um the program in the bottom figure allows us uh to catch out of memory kills uh just outside uh the box so uh it uses a designated kernel probe and uh it allows us to do that we can also count the CIS calls which is not that interesting you can count the network buckets and you can even take a look on the Firefox HTTP

Road Transport after it was decrypted and uh peeled from the TLs uh so all of this is just a tiny example and it's possible uh let's say in this example we can also create a we application firewall that says completely evf based just like the first firewall which was BPF based now with these examples in mind I want to look at an open source that I've created I called it SEC import and it's a library level Dynamic ebpf sandbox it enables us to confine specific modules in your code uh in the python process uh by the library or function level in real time now why because a library deres the Privileges of the of the process that

it's running under right I think it sucks um if we look at log for shell for example so a loging glass liary was abused to bind a session and open a server and it should never do it right it's just a logging Library it should only write logs it should never access the network the same can be said on XZ it can only it should only decompress and compress data and it should never run arbitrary system commands we'll get to that soon but in 2017 ebpf was not as mature and it was not present on any Linux machine but today it's not the case and 7 years later EF is literally everywhere and almost everybody uses it

so I I wrote this simple open source um that can utilize vpf uh to confine libraries in your code and um it uses usdt and Trace points which I've mentioned earlier and you can trace and then enforce the CIS of a given application to the modu level in python or to the package level in Python uh without risking your code of ever running it so it acts like a man in the middle between the python process and the Linux kernel and it's pretty interesting because uh it it does makes one day attacks less of an issue because you know that if one would have exploited the one day attack so it would not be ended uh um running so it

literally blocks the attacks and let's see how it works so this is an example profile you can see I hope you can see it by the way but there three different models each of them has different Cisco that I am allowing it to run so uh it's composed of three simple command uh the first command is Trace which simply traces a running process or a new terminal and Records all the CIS for every module in the code and then it results in this policy file then the simport build command compiles this policy to BPF code and the Run command allows us to apply and enforce the security on a running process or any one so in this example I've created a FP

application that simply has a back door in it I triggered this back door which is uh known only to the attacker for the sake of the example and just in time it violated my policy which resulted in six stop who stopped the application and as you can see the page did not return uh the the server did not return anything to the attack to the attacker sry and just to make sure you got it we ran the exploit in real time the sandbox detected the anomalous Behavior because the library Behavior changed in the Cisco level and then we sent a six stop to the process and we can you know you know just access the memory of the

process without killing it but of course you can also kill the process and um this is basically the notion of SE import now let's jump to the exit use case um uh SE import by Design uh should be able to detect XZ but I built it for python of course uh it was a PC for the notion of being able to detect behaviors uh that's changing in libraries in a given environment and of course import is not a production great solution uh it's merely a PC but it's working and it's amazing and spoiler alert we have uh detected the exploitation of XZ and let's see how exactly so which tool can detect zero days I mean theoretically none right um

so um if we look at XZ the attack resulted in a poisoned artifact in gab so um the attackers actually managed to propagate the back door the way to almost LTS versions of Ubuntu and Debian uh but luckily enough it was caught on time and when next the world has literally gone crazy I mean the back door C us with our pants down and I'm feeling very lucky and I'm feeling lucky and unlucky at the same time and I'm just thankful they not propagate all the way to ubun to LTS and that was discovered in time and all of this is just thanks to a curious developer that noticed half a second overhead in his

SSH sessions and um and I'm mostly concerned about what's uh what we don't know yet and what's only waiting to be exploited almost all the companies that build uh detection tools or visibility Tools around the world claim they can detect and remediate exit somehow and prevent it but they did not provide any evidence of doing so and I think detecting exed is not is not as hard as you think here's the secret the application context is all you need sshd is composed of many different libraries right among them is exad we understand that SSH should call lipes system command uh which can run up with record but inside sshd xet should never call this flow it should never call the

system command uh not in any real world not in any real world use case at least uh it is that simple how many of you were impacted by xed okay TR is have your organization approach this incident how long did it take to resolve it and why does it always happen during family weekends and family vacation am I right or what like it always catches us as I said with our pants down uh which which is exactly why I've originally developed and published SE import and how did we uh detect it so typical organizations usually rely on CV for vulnerability management but there is a problem with cves um they lack critical application context and and

they can also only be applied uh retrospectively so uh traditional tools would require a specific rule or specific uh cve in order to detect exed Library um vulnerability but we were able to detect it without any special rule or cve and based only on the library behavior um imagine SE import but wrapped as a scalable production grade product and fully autonomous and at oigo we build a vast Deb of runtime profiles for each and every Library out there that's widely known such as exit and SD we managed to replicate the attack provided by the samples by Anthony Williams on GitHub um we triggered the back door and expected the back door to be detected by the BPF sensor and as

expected our incident was created automatically and our evf approach which focuses on the library Behavior rather than a cve um caught it just in time and remember that XZ should only decompressed data it should never uh run arbitrary system commands even if sshd is capable of doing so um we don't have time for the uh video but there's a video on your website if you want to watch it and uh let me conclude so we've seen in these examples that evf can be used to identify and block application Level exploits in real time which cannot be detected in cicd or at build time sort was was just a PC for this Behavior based sandbox and in oligo we've built a

scalable runtime security solution that is completely production grade and uh detects application Level exploits in real time it can even detect zero days as I said and all of this is done in a much more efficient way and first of all uh stable way unlik before and thank you for thank you very much for your time now we have some time for questions [Laughter] hey thank you for the great talk so one of the last slides showed you that you are detecting the system uh um function call which is actually a lipy library call now on the other hand we talked about before that ebpf is good at detecting system calls not the system function but the system calls so I see

that there is a gap here yeah H just to make it more simple we called it the system function but it's actually fork and then exec V so this is what we we detect so so I have a follow-up question about that because one of the most problematic part of this back door was for from a detection point of view was actually that the fork and xaque is actually what an sshd server is designed to do so from a detection point of view from someone like you are sitting on the Kernel side and seeing only you know fork and xec V it is is the normal behavior of the application okay so so this is this is

exactly what is special on this approach because um this is what like what AI said that sshd is supposed to have like forks and and xx but how how you can detect inside what happens so what we did we added the application context so we basically know uh for each system call where it originated inside the application okay so so you can uh differentiate if it came from sshd and one of the functions that are really supposed to do Fork fork and xx um but but when you see that XZ is doing that you know that XZ was never allowed to do such a such a thing so this is this is the essence of the the approach very

nice thank you very much thank you do we have any more questions at this time you'll be able to approach uh the speakers uh during the course of the day and as mentioned you'll be able to find information on their website and in the future everything today will be up on the website related to today's um conference anybody else no in which case guy Abby thank you both very much thank you very much