Secure Containers: Do Reduction Strategies Fix Your Nightmares? - Michael Wager and Michael Helwig

thank you for the um intro so um we're going to give the talk together I'm doing also a bit um kind of intro to the topic and then Michael will present you the other Michael will present you most part of the presentation and um the Deep dive basically into the Container security so our talk is about do compon component reduction strategies fix your container security nightmares because now doubt if you worked with containers before you probably have some security nightmares around it at least if you took it keep a look at containers um let's quickly start with an introduction um Michael do you want to yeah sure so my name is Michael and I'm working at secure Consulting for Def secops and automation of security Assurance before I went to security I was working as a software engineer for many years and also there I was focusing on automation for quality insurance so yeah makes sense that I'm and was nice for me to switch to a security appsc area Okay um my name is also Michael Michael H I'm more on the side of strategic Consulting around building secular software development life cycles on death SEC Ops area um interested in all things security and I founded this small consulting company that logo which you see at the top Securo okay let's Deep dive into it so first we're going to talk a bit about what is the container security challenge um what what challenges do containers bring if you use them um in terms of security then we will Deep dive into one um solution strategy maybe for for many of those challenges which is component reduction methods and um this is basically what micro will also take over and talk about the distrus concept we will see a quick demo on how the distrus can help you to avoid certain exploitation possibilities um then we will go a bit into the the research in this area what is um what has been researched there on the market and and papers and so on and um arrive at a quick conclusion so um so what what are security challenges with containers we often come into companies do a bit of Consulting and we find out okay they are using containers but they do not really have a strategy for it yeah there is no real processes for the usage of containers so that means there is a lack of transparency into vulnerability scanning into vulnerabilities within those containers there is no scanning or there is some scans but no one looks at them um there are also no trusted repositories that means there is no limit on the base images that companies use you do not know where containers originate from where base whiches are pulled from um no control No Limit so um a lot of security headaches uh at the same time containers are everywhere so even your cloud services may run on containers you get containers from third party vendors as a delivery as software deliveries for commercial of the Shelf applications um so basically there's a lot of places in your organizations where containers might already may already be in use but you are not really aware of it potentially um at the same time with containers there comes a bit of resp responsibility shift because containers now suddenly lead to developers taking over a larger part of the product Stacks they suddenly have um OS libraries or S parts that they are responsible for and they are not used to that let's say it like this yeah so they are not necessarily aware of it they built the application they put it in a container they ship it everything seems fine um but they are not necessarily aware of of all the baggage of all the overload that they um have in that container um and one common reaction that we also experience unfortunately when talking to Developers then is a reaction it's not our codee yeah so it's it's um we do the application we just put it in a container but the rest is not our problem we are not responsible for the vulnerabilities inside the container um but Ops is also not responsible so who is in the end um you do have complex attack surfaces with containers um we will not deep dive into this but just thinking about it you have the application layer you have OS layers you have configuration issues potential network communication container Networks um that might be interesting and you have hypervisor attacks so you do add a lot of complexity when using containers and another thing is the more different base images the more containers you use the more headaches you will have in the future because security is something that does not stay constant security degrades over time there will always be no uh new vulnerabilities new issues appearing and the more diverse your container landscape is um the more headaches you will have in the future to keep all that under control um if you look at statistics about vulnerabilities um this is some this report is not from us it's from from a vendor called cystic but we found it very interesting um you see that 87% of images have high or critical vulnerabilities um 30% just have low medium or no vulnerabilities so there is a large number of images out there um I'm not 100% sure how many they they scanned I read in the report the number was at least 25,000 but um there are other number so maybe check out the report yourself if you're interested um and 7% have higher critical images uh vulnerabilities and only a small number of those vulnerabilities is actually exploitable so they said like 2% is actually exploitable of course you don't know which ones those are so what does it help you to know that it's only 2% if it's inside your containers um but the other interesting statistics was that um there actually is a a patch available for many of those fixes and for many of those issues that means um 71% are patchable it's just not applied in the base images so so um the base images are not up to date um but only 15% of those vulnerabilities are actually in use um of the affected components are actually in use um from the application at runtime that means you have a lot of stuff in those container images that are that is not used um but that contains vulnerabilities and um yeah so what do you do about it as a security team it's quite straightforward you look at the oest design principles for example and you find hey the more software I have the more complex my landscape is the more vulnerabilities I will have so I'm looking into minimizing my attack surface and this is basically one of the strategy that we also explored in our company to bring to our customers to tell them okay what can we actually do to minimize the attack surface on containers and to make sure that there is not so much overhead for applications that is not needed but just introduces headaches for the security team um okay yeah and that's it and now I'm handing over to Michael to show us basically how that can be done yeah thanks Michael so I want to start with the sentence it's secure because it's running in a container so we get that sentence a lot it seems like still the misconception that if you throw something in a container and maybe it's kind of true because processes are isolated from each other and you have that Linux kernel features uh making up that concept of a container but still there is something running inside the container and and I I really love that illustration of it so just as an intro to tell you there's still a lot of things can go wrong and also we get another sentence from stakeholders especially stating that teams are not allowed to select any base image they want but I I want to compare it with open source components and the image illustrates um for example maybe you heard about lock for J was a critical vulnerability in a Java open source Library supporting logging and uh in the end hackers don't care with which part they take to exploit your application and that's why we we compare it with open source components so teams are allowed to select any base image they want and that's why you responsible for the security and uh of your selected base images and this is why why it's important that we scan them guess you all know security scanners container scanners are scanner tools who statically scan your images for non vulnerabilities they also find stuff like Secrets or uh credential lying around in the container and they are very easy to integrate in into cicd pipelines so you can easily set up something like not allowed that developers are not allowed to merge a put request if they are security findings because everyone's talking about shifting left and automating security assurance and at this point a little story from our experience working uh Consulting in a centralized product security team so imagine lot of more and more companies are moving to the Cloud we're talking about Cloud native and and then they have like 10 or 15 years old Legacy Java applications should now get containerized right and you have these management policies dictating that they are not allowed to deploy to production if they are critical or high severity findings and then imagine us coming and scanning these applications 10 years old and they have like 200 findings only critical in high severity which is often in our experience that WTF moment for the for the teams coming to us like t Security team first uh we want to deployer in two weeks and second we it's not our like Michael said it's not our responsibility and not our code so this is why this is one of the reasons why we why we came to all this component reduction methods as we um call it and what it what it is is basically um a tool um like tools who minimize the components lying around in your container with the goal of minimizing the attack surface and the noise of container scanners and there are four important tools available which I going to present now so first we have Google distes it's an open source project by Google and they made the bus word distes I guess some of you have heard about it famous already 2007 and they're providing some production ready images for certain runtimes like Java NOS python go and they're very small in size compared to regular container images like the static image is just only 2 megab which also nice side Advantage then we have redhead Ubi micro based on redhead Universal base images and the nice thing here is that you get they don't provide so many prod ready images you have to build yourself for certain run times but you get the same Linux um security hardening and uh response team as you get with any redhead Enterprise Linux and funny thing is that if you Google for distes you will you will certainly find a blog post from redhead bashing against Google distes and you read it and you think like okay maybe it's not the best but if you read it until the end it's just an advertising for reded Ubi micro so then there is Ubuntu chisel it's pretty new from 23 and highly inspired by Google distes um it's open source project by canonica the organization behind Ubuntu and they're also not providing so many production ready images you also have to build them yourself um but a nice thing here is that you have the Ubuntu policy behind them um making sure that all critical and high severity vulnerabilities are fixed within 24 hours and finally we have chainu guard images the chainu guard is a security vendor founded in 21 and the interesting thing here is that they're providing images for all runtimes even maybe not so popular ones like Ruby or PHP so and also they really have zero vulnerabilities for for all the runtimes they provide so will present that in a second but for now coming to a little live hacking demo showing you like an example of a vulnerable application and how distes could prevent these classes of attacks so let me see if that works yeah I made a movie so to not blow it up to IP addresses okay what you see here guess it's should be large enough right is uh a no sh application simp simple application just um starting a listener um acting accepting an HTTP request on the route and if there's no query parameter it just sends an HTTP response no query parameter provided but if there's one it will directly execute a system command with the input of the query parameter which is of course very bad example but there are more complex implementations coming down to exact this issue right so just to to demonst stra and it will send the standard out of the system command back as HTTP response so we want to check I already built it simple Docker file um it's based on node version 16 Debian Busey and it just basically puts PS over copies over the source and installs its dependencies exposing the port and run it so let us try to run it can see in the bottom and now it should be up and I want to test it with with curl so let's just fire simple HTTP request without any query parameter okay application is up and running so let's let's try some and as we know it executes a system command let's try some sh commands like list directory and at at that point I guess you already get the point we get we get uh directory contents from inside the container from the outside but I want to present a little more sophisticated example so there's a remote machine I have a net cut listener on a port 4445 and I want to exploit the application now with getting a reverse sh against that machine so and as netcat is is inside inside that not 16 image I'm lucky and so this the payload will be a little bit more complex and I blurred out the IP address but basically it connects against the IP of my remote machine with the port 4445 injecting bin bash and to get the reverse share connection and if you fire that HTTP request you will see that no response is coming back so line 14 here in the code will now connect via netcat to the remote server we get the connection in that machine and now we can of course try it's working and we got a reverse shell here so who am I we even root so you see we from the outside we got a sh share connection inside of the container and it's it's as it's root like maybe you know it with Docker if you build an image it's it's unfortunately default defaultly built as root and the root inside of the container is the same P ID as on your host which is which is a real issue with uh that's why it's a best practice to always build explicitly as user declarations to your Docker file so not build as root so we can do a lot of eneration here like we as we root we can install stuff I guess you get the point so now we just want to show how dist list could prevent these kind of a taxs in a very simple way let's have a look at our Docker file and in Docker you have a concept of multi-stage builds and in the second stage as you can see we're just switching to the the Google dist toist base image for njs version 16 and copy everything over basically and then we expose the same part and run it so first I need to remove the old machine and now I'm going to build it again and as you can see already it's fetching from Google Cloud registry build is done let's run it and test it with curl again so let's test our list directory it's not working anymore okay let's let's test our so the application itself is running fine which is quite important yeah just uh it's running with this less um and let's try our more complex payload again and it's not working HTP uh response is coming back directly and we not don't get any connection on the remote server and of course we want to see what why is that so let's quickly have a look at the locks just need to find the ID of the container and now we can have a look at the locks and I just have to blur out the ID IP but in the top here you can see that it draws an error that there's no shell so basically that's the solution and also at the for the end if we had scann this with trivia which an open source container image scanner we would have for the old image we would have 238 vulner public vulnerabilities of which are 221 high and 17 critical so imagine the team coming to us like how should we fix that um and if management dictates you are not allowed to deploy to production with that so after you switch to this L you only have two of s twoo High so pretty nice outcome I would say okay then coming back to the demo we also think that it's a good idea to have scientific proof for what we say and with CIO we have a long running collaboration with the University of applied sciences in alburg and lately we joined a little research project on container security and we want to have we wanted to have a look at three questions here first we want to see if these component reduction methods really drastically um remove the the noise of container scan findings also we want to see if typical findings are in actually exploitable like exploitation probability and also third we want to see some Implement implications when integrating these methods in into the development life cycle and we are in the middle of the research but I can present already a little bit of the results so for example we scanned five uh we scanned like 20 image or so all from the presented methods I I showed you and as you can see for example if you have a look at the chain guard images or the distes images or for example the Ubi micro ones for Java they all have like zero findings really really zero this is also the the thing they advertise with for example chenard yeah so this looks good for research question one and for the second we we looked at we in total we found uh 563 cves and besides the besides the csvs score which is typically used for remediating vulnerabilities for critical and high there's also from the first organization the epss which is the exploitability exploitation probability scoring system and we check that API for all of the cves we got and we see that only one has a and what what the score says is that it's the exploitation probability within the next uh 30 days and they taking much more sources into account like Metasploit exploit DB GitHub advisory database and much more to calculate uh the score there's a nice um paper explaining it and we only see one cve with with a probability of 38 one with around nine and all the others of 500 are like under 5% which is prettyy interesting so exploit regarding at least according to that score exploitation probability is pretty low for most of the findings so I hope you feel like that after you deployed the dist this image coming to the advantages like we get minimal images only containing the runtime and the application and its dependencies um no shelles no package managers no no other stuff no maybe no netcat binary which your application doesn't need anyway and therefore reduce attack surface and less findings of container scanners also Prett it removes entire classes of attacks like like I thought in the demo and of course the side Advantage here is that you get faster transfer times and Less storage size which also leads to less costs which also is a nice business case you can argument when you want want to integrate that and we get faster build times which is also always a good thing regarding def secops and coming to the disadvantages or rather challenges so we have complexity and compatibility issues so to get let me give you a quick example so we recommended this to L for no CH application and they tried it and they got a runtime exceptions coming back to us and say like hey security team this is not working and then we we had a look at the application and saw that they are dependent on on a Kafka Library which is in turn uh um a rapper for Plus+ Library which has the dependency on a compression shared object binary lying around in a normal Linux distribution which was not lying around in the Google dist drro and after some fing around with ldd we get the dependencies and we found out that you can just install in the first stage the shared objects you need and you SEC you copy them over in the second stage which is quite uh which seems to be the accepted solution and uh the thing is just this is a little bit complex and that's