Context-Based Security: What Your Cloud Native Apps Really Need

so ladies and gentlemen uh to the final uh presentation of bides 2024 um we'll hand you over straight away to Ben hburg hey everyone I'm the last one for you today um just to start up this talk I know everyone is tired already uh you know it was a great day but you know I'm sure that if you give you gave full attention to all of the talks now you're are already tired uh um I will try to make it interesting for you um and and um yeah uh let's jump into that so let's go back so a few words about myself um I'm it's hard to say today but I'm nearly 20 years in the

industry I actually finished my um BSC not really far from here in UDA uh so as you see I'm Hungarian speaker as well um so um I'm been on many sides of the Cyber industry both in the offensive and um defensive side product and so on today um I'm both an open source maintainer of of Cub escape and the CTO and co-founder of a company called armo um and I'm I have to tell you that again on a personal note um I'm usually giving like I would say between six to eight talks a year never had the chance to be here in Hungary and in Budapest so I'm I'm like really excited about it uh and

I will try to put a little a few Hungarian words into the presentation hope you'll understand so uh just a few words because about the um my project and my company because it is really related uh to this talk so Cub Escape is an open source project actually a kubernetes native uh uh security platform um which we started to work on three years ago uh which was actually um you know uh sometimes open source projects are coming from different interesting places we started to play around with uh nsa's uh um recommendations about how uh how to secure kubernetes clusters and it turned into a script and the script turned into a a go CLI and a go CLI from one day to

another turned into a a viral uh GitHub project uh which was a really good thing and you can see that as of the GitHub Stars uh we are advancing um really fast in the last two years by the way if we are here already uh how much of you are working with kubernetes out of the nice okay um the second is armo my company uh the reason why I'm mentioning beyond the obvious is because uh armo has contributed a lot of data for this presentation um so thank them for that uh armo is a cloud native security uh company uh covering uh both from runtime detections and the posture uh part of the cloud native security landscape so

um yeah it's a good company come and visit our website um so usually you know all the offensive cyber talks are are like those very um damaging because after these talks where you hear that 60 seconds someone can Elevate the Privileges from a simple user to admin user you are thinking that we are doomed and we cannot do anything about it and it's like kind of a problem okay that where our world is and you know all the the computers are just world is just growing growing growing and we have to you know us as an industry we have to keep up with all these changes but I I really hope that today I'm going to you

know give you a little bit as a relief for the end of uh this day I'm going to give you a little bit a different perspective of how can we um how can we use different data streams uh in order to get context to vulnerabilities we are managing and how to understand that how we as an industry can focus on the things that really matter and try to you know uh uh tone down uh the noise ratio so uh we'll talk today about um about container vulnerabilities by the way from the from the general perspective it these are not really different from any other kind of vulnerabilities these are vulnerabilities um the only big thing

from a from our perspective is going to be the attack vectors of how an attacker can get into uh attacking containers we'll talk about it in a few minutes later we'll talk a little bit about reachability analysis of vulnerabilities um and and later on we'll talk about more advanced um uh prioritization uh parameters we can use so again I'm sorry that I'm surveying you through um this talk how many of you are dealing with vulnerability Management in your current sayate one one and a half yeah you guys do don't count um so okay so let's let's talk a little bit about how the vulnerabilties in container world so um containers are really bound to the I would say cloud

technology landscape which is by the way is not really true I'm as a Linux person um um you know container for me containers were around ever since Nam spaces were introduced into the Linux kernel um but in but from an industry perspective containers are really associated with with uh workloads in uh in um Cloud environment and therefore two there are two major attack vectors when looking at containers one is of them is supply chain supply chain is everywhere um supply chain is starting to be the same thing as fishing like we always have to talk about it we always have to remind ourselves because we tend to forget it but then again comes a vulnerability

like what we seen this morning the exit vulnerability and we we understand that you know people are actively trying to um temper with uh the softwares we are building our systems on this is One Direction the second direction is obviously the N direction of network so Cloud uh again containers uh are are accessed by the end users uh through Network there there is less of concerns of attack vectors where the uh attacker can physically access or get into the vinity of of a container so we are not really concerned about someone breaking into the data center of AWS and uh you know trying to connect you know one of the blades or tap into the physical Network there or something

like that because we understand that or at at least we we think that Amazon is trying to take care of this pretty good and maybe since of their uh uh knowledge they may have you know an advantage there to do this better than maybe we might would do it so therefore we are left with the uh the attack Vector of application exploits and uh um when we're talking about application exploits there are obviously the application security perspective which is well known we are when we are uh um in our in-house developments we are releasing software we are we are or we are developing software we have to make sure that our developers are not adding

vulnerabilities in their applications and so on and it's a it's a well-known part the second part is that we have to understand that our Even in our in-house built software but in general other software components we are using are also uh built on open source software or public software and we have to understand if there are vulnerabilities inside so therefore um this whole um uh practice of of scanning for vulnerabilities in containers or in beforehand in in Linux host was born let's understand in our security posture what are the vulnerabilities we have now the the problem with it is that when we are running these vulnerability scanners for containers we tend to have a lot a

lot of vulnerabilities and these are like just numbers that sometimes user cannot hand handle so to be frank if if a container image which is running right now in the production we have like more than 100 of vulnerabilities like it is something that it's really hard for our organization to manage because what do I do with 100 vulnerabilities how can I fix like all of them and most of the time you cannot even fix all of them because there are like vulnerabilities that are don't even have fixes there are updates which are breaking the application and so on so it's it's it's a really really messy thing and this is what uh um the state

of uh of vulnerability management we call it a CV shock so CV shock is something where um and I have to say just again anecdote this guy stole this from me um uh the term CV shock so um uh CV shock is is a state when when you don't that simply the organization decides that not to handle vulnerabilities because there is no no point like they're working hard to remove vulnerabilities and like a day after or three days after like there are new vulnerabilities out and reported against the same software packages and they just don't know how to handle it and because it's like a vicious cycle of of like fixing the discovering fixing discover

now um if you look at and again here this is something I did like an year ago like I counted like in this well-known uh uh op Source software I counted the number of vulnerabilities they had in their official releases the latest versions and when you have for example you have nearly 200 vulnerabilities in a Kafka release what can you do like this is the official latest official release and now um you know there are multiple ways to uh um to handle this problem because on the one hand obviously um you know if you have a kfka has 200 vulnerabilities in the container image then we have to ask ourselves whether really these 200 vulnerabilities can be exploited by an

attacker because the vulnerability finding is does doesn't equal to the the fact that whether this vulnerability can be exploited and there are multiple reasons for that um maybe the the code path which is executed doesn't touch as the the specific uh vulner vulnerable code therefore an attacker would never be able to give an input to the uh uh to the software which would then cause an an exploit on the other hand uh um there is maybe it's a dormant binary inside the container because devops loves to put different nice need tools into their container images because they know that at some stage they will need to debug that software and when they the day they

will need to debug that software they want to have bash inside and they want to have busy box inside now I guess everyone of you because we are in a security conference everyone who hears the world busy box like understand all the consequences of it so again this is obviously a decision of of an organization level and and compliance level what what they need to treat or what not but there is something for sure and this is what is called reachability analysis and what used to some we at ccape used to call at the beginning uh relevancy because back then like a year ago year and a half ago reachability wasn't uh that that uh this term wasn't

really leading um the discussions and the thing we said that whe again whether in inside the container whether a com software component or a code which is being uh uh part of the container image whether it is executed or loaded into the memory during the runtime because if we say that it's not so the the part of the uh uh the software package or or or even a subset of uh of a package is not used or not loaded into memory that means that it cannot be exploited and um here um here is a very simplif schematic drawing of of what our open source project does what it does is uses BPF which was fairly mentioned this morning um uses it

monitors containers in a kubernetes um cluster containers in kubernetes are organized into something called pods um and what it does it actually follows the file activity inside the container during the execution and think about it as you know observability tool which is some simply list out all the files that were touched during the runtime of a container now on the other hand Cub Escape scans the images using classical scanning methods and get a software bills of material or other words as bomb sbom is simply a list of you know what software packages are inside the installed inside the container um I'm not going to more deeper because we don't have a lot of time but there

are a lot of discussions around esom but the something you have to know that the way that most scanners are working they are trying to find what kind of software packages are inside installed inside the container and put them they're they're listing them inside this sbom document and what we did is um we took all the files that were the list of all the files that were touched inside during the runtime of the container and cross reference them with the sbom the sbom lists not just the software packages inside the container but what files belong to that software packages and we could Mark all those software packages that were in touch during the runtime of the

container which is then we can say that these since these software packages are not actually touched then they are not actually loaded into the memory then they are not really reachable by by an attacker or not relevant for uh uh for uh from a vulnerability management point of view so we got the list of relevant vulnerabilities um just again a few words about how the uh vulnerability scanning Works vulnerability scanning work as I said first at the first step what it does it uh uh creates a self-ability of materials and then it cross references uh the software packages and their versions against wner bases so for example if we've identify that there is um I don't know um curl package inside

container image in a uh uh in the version of the uh red head distribution then it will go to redhead vulnerability uh uh information it will check if what are the vulnerabilities belonging to this very specific curl version and uh uh and add them to the vulnerability list so what we are doing here is actually we are shortening filtering out the unused uh uh packages from this as bomb and therefore the vulnerabilities list of vulnerabilities will be also uh filtered out now as a result we got like very uh for me very impressive results think about it let's check I think Kafka you can see the relation of number of vulnerabilities be uh before filtering and after filtering see

about redis en genix and so on there is a huge huge huge discrepancy between what a static scanner can say about the vulnerabilities inside the container and what you know dynamic analysis can help you and I'm returning to the title of this uh uh presentation uh context matters so under when you're managing your securities relying on you know a single data stream can be very misleading and lead you into uh uh into um a lot of you know work hours burned just for nothing I can tell you that that from as a rule of thumb um we are removing this way we are filter filtering out vulnerabilities in between 80 to 90% of uh container image

vulnerabilities this is by the way this is very different from container image to container image there are like scratch images and so on where where uh um which are I would say clean images but in again as a rule of thumb when we are managing huge clusters and huge set of of container images we don't have effect on this is a huge gain I need to mention be that there are more Advanced reability analysis tools like the guys here from uh oligo doing that what they are doing is actually they are not just at the level of the software packages are able to filter out but they are able to filter out um at the function level

right uh uh um packages uh and it it it can even more reduce you the number of vulnerabilities you can uh you have to manage now I will bring you um three other um known and less known prioritization um um um metrics one is them is CVSs I guess here everyone knows what CVSs is it is the way of Simply uh giving an information or um communicating information about vulnerabilities and tell and give basic information to uh uh to the users and understanding what are the effects or how a vulnerability can be exploited or what it can uh what it can the effect can be um I've never heard someone who was satisfied with CVSs um it is uh I've just two month ago

I had the privilege to attend first uh first uh uh uh conference about how to improve CVSs and the fun part was that they didn't find a way to improve it uh uh but they decided that they will uh you know will try to work on it more it's not that bad the problem is that there is like uh the most of problems around CVSs is actually there is no standard way to uh uh to make these metrics so if there is a vendor who is deciding that giving a specific score speci attack metric it does it because of there is no it does it in a way that he think it's right but if uh another

vendor decides to get uh to Mark and and rank his own vulnerability we might have a different idea so what does it mean to have I don't know uh um uh uh whether it this W affects Integrity of the system and at what level so there is like a there is there are some problems here obviously but it's it's it's the thing that it's everyone works with so I the thing I wanted to draw your attention here is um the part which with we started to talk about is the the attack Vector in case of containers so I told you that there are two main attack factors one is you know supply chain uh the second was through

Network one of the uh uh prioritization factors in um in CVSs is actually the way the where is ATT VOR how an attacker can exploit of vulnerabilities so there are four uh um levels of of attack attack vectors one is physical the attacker requires a physical access local the uh uh the attacker needs to be on the auto on the same machine more or less agent adjacent and network so the first two I mentioned uh requires physical access as I told you in the cloud world this is Le less of a concern for the users the network is the most concern because it means that the exploit the exploit can be done over Network and the the

adjacent is is not really a good uh um attack Vector because it would still require to be require some kind of U I would say more closeness for a physical perspective and it's and it's also not really a big concern so when we are prioritizing in CL in the cloud world uh vulnerabilities we are mostly concerned about Network now from our data sets of uh of our armos customers we saw that only 21% of of the vulnerabilities we are seeing in our customer sites are network exploitable which is like one fifth uh a very small not not very small a small uh uh part on the other hand we've evaluated as well how many of the

workloads has uh Ingress access what it means that how many of them are getting uh uh traffic from the outside world uh and it's only TW uh 11.8% um maybe something like 1/10th so all the other workloads um doesn't get direct uh uh uh access from the outside world so I would say they are like third or or fourth in the on later in the Hops from uh the external networks so if you're thinking about this two you will notice that if you are taking away all the I would say 79% of the vulnerabilities we are seeing and out of the what we left which is the 21% we only need to look them on those

work clads which are actually getting traffic from the outside world because if all the others are not really exploitable so and again guys I know and I'm sure that someone will ask here that okay but what if an attacker is able to I don't know put here and there yeah you're right I'm talking about how we're managing vulnerabilities and mass okay not like very very speciic thing but how we are uh managing like I would say based on the 80 20 uh principle the vulnerabilities another very relatively new and uh uh important metric is cev or non exploited vulnerabilities cisa is uh decided that there is no standardized way to uh uh to communicate uh threat intelligence there

are obvious well-known sources of threat in intelligence information commercially they are available but this information should be free and shouldn't be limited only to to Comm to the commercial world and therefore if I want to know if a vulnerability is actively exploited in the wild there should be a government agency in the US to communicate it and they defined uh uh the non exploited vulnerabilities database um it at the beginning it was like I would say 90 something per of those those were Windows related vulnerabilities less prevalent for the cloud native World um but at the later but lately in the last year or so also um this information started to uh flow into the cev database and you can you

know download this tomorrow from the website of of cisa and you know start to look whether the wner bilities you have in your environment whether they have known exploit uh in the outside world um just again to give you a little bit of of uh feeling every line here uh uh in this table is a different uh customers of ours obviously we reducted uh uh uh the names of the customers but what you see on the left is the total number of uh of vulnerabilities they have in their systems unique uh and on the right you will see how many of these vulnerabilities are in the cev database um by the way it's not um um not in the

presentation but I can tell you that obviously there is a huge overlap between those numbers so these are very common vulnerabilities that have been that are around in different systems um so yeah so you can see that there is a huge gap so out of the 23 uh uh uh, vulnerabilities of this company have only 40 them are knowly exploited by the way which is a lot uh don't get me wrong okay this has to be something that has to be handled um but again if you see at the relations I you can already understand that how bad the outputs of vulnerability scanners are um the last but not least is epss um again it is relatively new

newish um EPS is a um um another scoring method uh by nvd the national vulnerability database and um it is a machine learning method they tried to say the following I have a vulnerability maybe it's not exploited in the wild but it might become something that it's going to be exploited in the wild in the near future I don't know when and and if can I predict whether this vulnerability will be exploited at all now there is no the way this uh uh scoring system works is not binary so it's not that yes yes it's it's going to be exploited or no it it assigns uh a number between zero and one uh and the

more you're closer to one the more uh um chance that in within the next 30 days this vulnerability is going to be exploited how they did that um cannot explain to you uh uh uh because I'm an engineer originally and I don't believe usually in this kind of magic but uh uh but I have to tell you that the fun part is that actually it's it's quite neat we've invested a lot of research in into that and actually we were really great correlations here and it proved me that it's it's uh it's not a bad bad idea at all they are taking all the descriptions of the vulnerability entry and the scores and everything they know they fed

into the machine learning engine and it emits this number this number is being updated every day so you can every day you can download the newer versions uh and this will change in the time because again it tries to predict whe whether the vulnerability is going to be exploited in the next 30day so the the way it it it works so the way it was engineered was actually try to they weren't trying to be very deterministic about whether this vulnerab is really going to be exploited what they try to be is make the prioritization of our work much much better so and this how uh it's going to be much more interesting for you to

understand this drawing because I think it it really uh explains uh the the design idea here so they said the following so you you have to you can take CVSs uh uh 3 and and you can say Well I'm going to handle all the vulnerabilities in my system that has 8.8 or uh score or above by the way 8.8 is a pretty high score okay so you should be looking below as well not just above 8.8 but but and they said that the effort is like I need to handle 253 vulnerabilities out of a thousand this this big circle here and at the end only this small circle the the reddish pinkish Circle you see

here is going to be those who are really exploited so what they said that if I'm investing you know this amount 50% of my uh um uh uh time in order to just to touch like 5% solve these issues right because I didn't solve these issues these are going to be exploited afterward anyway I've invested this time into solving this it is a 5% efficiency okay because I've this was my investment and this is what I fixed and it it really mattered so designed epss v.1 and when they said the threshold and again I I'm trying to remind you that we are talking about not a binary scoring system but uh but a fuzzy log uh uh

system they got to a place where they said okay I need to fix these vulnerabilities based on this threshold again I'm getting to uh around 50% of the coverage of the really exploited vulnerabilities and I got to 12% uh uh or or 13% efficiency in epss V2 which was much better and this is what we are using today uh the they said that I'm only handling all those vulnerabilities that are above threshold of of um 1 uh 0.149 and this means that I need to fix 47 vulnerabilities out of a th000 again I'm still covering only half of the vulnerabilities that are going to be exploited eventually but at least I was efficient nearly much more efficient

than all the cases below so if you look at this epss doesn't say that I'm going to show you all the vulnerabilities which are really going to be uh uh exploited but I can really put you in a place where you have to deal with much less less vulnerabilities so um again uh why we are using it the reason why we are using apss it is because uh it really helps us to unload some of the work while keeping the same level of uh of uh of efficiency at the end of of the vulnerabilities sorry not efficiency but with the same outcome remove uh improve much more the efficiency and if we're combining all these

vulnerability uh uh prioritization methods or filterings which I told you before so if I had like around you know uh 200,000 uh uh sorry 20,000 wner abilities and I'm only you know taking out all those which are uh uh not in use and I only looking at who are getting traffic from the outside world and which of them are exploitable and which of of them is going to be exploitable in near future based on our our our our test we are seeing again amazing numbers of around 2 uh8 uh sorry 98 99% uh uh vulnerability noise reduction this is like uh for an example of uh of our demo cluster we had when we installed it we had in general we had

like nearly 3,000 vulnerabilities and you can see that you know some of them are knowly exploited some of them have a like likelihood which means that it they have high epss scores and but when we applied all these filters we only left with 17 vulnerabilities um and these are like all of these are based on open-source uh information uh nothing proprietary here uh obviously the kubernetes context is is also information you can get in your local environment as well so it's it's a very very huge uh uh uh component in noise reduction and uh um this is why I saying that context is matter if you can if you look at it we took EP uh uh ebpf

as an input to understand uh uh what is loaded into the memory we took different prioritization algorithms and we took also the actually runtime context of the container into uh into the computation now I wanted I don't know do I have more five more minutes so how about tan onp no uh uh it's the icing on the cake I think I think cherry on top I look more cherry um so I I will tell you uh uh something else which is also I think it's again it's very uh um not related to uh vulnerability management but in general to to hardening of workloads and how ebpf can be uh uh observability can be used to improve

security so one of the um in the container security World there is the setting when you're running a container whether you give uh run the containers in a privilege mode or not it's by the way usually it's it it's the shittiest configuration you could ever have like it it bypasses everything lsms in the Linux kernel like it's is the usual stuff when you know when Docker started to Define all these apis and said okay I need to like let's do the first version and the first version stuck with everyone and like you know today we're it's still still out there um so privileg containers are very bad privileg containers are effectively can do anything on on the host

um and um one of the you know one of the major requirements is to in uh in kubernetes environment is to remove and not to run workloads with with the the privileged flag on so when you're running a scanner it will say okay you have a uh you're running a container and privileg true like remove this like don't do that now what's the problem with it the problem with it that you know someone reads it okay let's remove it but the but the interesting part is that there was a reason that why it wasn't there in the first place right because uh it is not the default setting uh someone gave a privileges to that

because most likely because this is how the application works it it requires PES to do something um and um what we did in the Cub scape project is um again we used our ebpf sensors to uh understand what are the system calls uh and what are the um Linux capabilities the container uses and and it enables us to understand whether a container is really using these uh uh uh authorizations is these privileges or not and this enables us to tell you as a user okay either well you know it's indeed your container doesn't use this privileges or uh uh you can say well you have uh then you can obviously remove uh uh this flag or we can say to

you look it uses them but you can you know replace them with much more fine uh uh tailored configurations by adding like very specific capabilities your application needs and and not to give them a a cart blanch or or open ticket to do everything in your environment so uh um it I wanted to add this like very thing last thing here to to show you that that like the power of this security observability um just opens up so many capabilities and to make the security much better um and and you know I I don't know how you but if you want to contribute to the open source project or want to be in want to try out these

things okay we are Al always open to people who want to to contribute um just to Cube Escape but in in in Cloud Security in in general okay uh that's it that's for now um I will ask just for formality if you anyone has a question but I guess you want to go home already uh or or to have a great beer but anyhow is there anyone with a

question okay uh awkward silence uh thank you very much you can reach me anywhere here on the Discord in slack and thank you very much thank you very much ben so and thank you all uh as well for your attention uh I'll hand the mic over now to ingred who will uh give our closing remarks I am going to be very quick as well because you chose the beer so you must be really tired I'm tired too um thank you than you so much again for coming really I hope you had a lot of fun today and I also particularly would like to say thank you to our speakers some of you are still here because I

think the lineup this year was like really really amazing so thank you for coming as well and I hope you had a great day guys thank you [Applause]