9001 Ways To Break Out Of A Container - Josephine Pfeiffer

great thank you everyone for coming so um I have to spoil this right away um there won't actually be 9,1 ways covered in this talk this was clickbait um but yeah thank you all for coming I'm Josie and I work as a consultant at red hat and yeah I'd like to take you on a little tour of some of the fun ways that we can uh exploit containers so um because will be talking about like a lot of low-level Linux Concepts um this talk will be also kind of dinosaur themed to keep your interest and every now and then I'll do like a little recap of the things we covered in dinosaur terms so yeah hopefully you'll

not fall asleep um yeah a little refresher on container Basics um the way most developers will most likely interact with containers is just through a simple text file so Docker file or in the more vendor agnostic term for it is container file and yeah it's meant to package your code any files you need for it to run and its dependencies libraries in a way that you can run it anywhere that is the um the the idea behind it really so the will benefits it's sort of promises over using VMS or something like that is that it's supposed to be yeah for portable scalable fast and isolated so there's a little bit of foreshadowing here um we

we'll see how isolated they really are and in more lowlevel terms just like there's no Cloud but just other people's computers there is also no container really it's um if you run it on a system it's just another process that's running on the system so as far as post separation mechanisms go um you have these name spaces basically all the yellow boxes on top is one container and those are meant to isolate the various system resources so like the process IDs network interfaces and mount points and um then that yellow box is capabilities and those just assign specific privileges to processes without granting full rout access so so an example for that would be maybe binding

a privileged port for a server or something like that and then you have the c groups and those manage and limit resources so like actual um Hardware type resources like CPU memory or IO and um yeah they limit how how much of those the processes can actually use um in between we have the container runtime and the container runtime kind of manages the life cycle of these containers so uh starting them stopping them removing them and they do that by interfacing with the kernel and um they also sort of provision these uh name spaces and capabilities and c groups required to run

them great and then we also have um the security modules that um help um in that regard so security modules in the Linux kernel are like a framework to sort of provide a standardized way to plug in um you know additional security uh policies and access control mechanisms and yeah they mostly focus on files and system resources and then at the bottom of course you have the hardware that you're running on um there are some interesting topics there as well to be discussed like around confidential compute and confidential containers as well where you can run containers with encrypted memory in the likes but we'll ignore that for now all right and in Jurassic Park terms

the hardware is the park that we're playing in then the kernel um parts are like the guards or the control room and then the individual containers are like the little dinosaur enclosures all right let's take a look at the attack factors you um have in in those containers so um a lot of them can really be avoided by configuring things correctly like many things um the container Escape stuff we'll be looking at today happens in this purple box really so it's at the intersection of the host OS because what container Escape really means is that we break out of the container and take over the host resources and yeah it kind of uh intersects with um the container runtime

the host OS and um the container itself um so those are the major types of container escapes so the file system ones are when processes gain access to the host's root file system and that can well most of the time this happens through a a misconfigured or overly permissive bind Mount um I'm like very comfortable in the kubernetes world so this is something you see quite common uh commonly in like the kubernetes world where people just use stars and their AR back policies and yeah that's uh not good and um for memory um you can cause a buffer overflow in a containerized process and then uh manipulates shared memory regions and the container run time

itself if you misconfigure it you can uh execute commands directly on the host system and bypass some of those boundaries we saw earlier and the most interesting part in my opinion is the the kernel type escapes and um yeah it's uh mostly around gaining elevated privileges on the host and then bypassing isolation that way to uh gain direct access to the host system and we'll take a look at one uh particular practical case of escaping uh the kernel uh n um so yeah ebpf uh was originally designed just for packet filtering so the BPF is Berkeley packet filter and it was later extended that's what the E is for um to also allow running user space

code directly in the kernel without loading lkm drivers and the sort of marketing tagline they used to use is that uh it does to the Linux kernel what JS does to HTML um but they kind of cut that out probably because JS is kind of a bad look to be associated with nowadays and yeah the major benefit of it is really that developers don't need to know how to write kernel code and it sort of lowers the uh barrier of Entry to um running things directly in the kernel which is also consider considered safe and you get um performance advantages of course and um is the most popular and wellknown use case for ebpf right now is

maybe cium I don't know if you've heard of that um a cni for kubernetes and um also Swiss company and recently acquired by Cisco um so the attack vectors you have in ebpf are on one side the ebpf verifier which um um you can exploit by just tricking the kernel into loading unsafe bite code and you can use arithmetic and logic errors there's the just in time compiler that you can uh also manipulate with errors and then finally just memory corruption so the particular case will be looking at uses the ebpf verifier so it's meant to sort of Ensure the Safety and Security of ebpf programs uh before they're executed in the kernel and it has operates in two

steps and you can kind of think of the verifier as the electric fences in Jurassic Park and they're uh designed to keep the dinosaurs from running wild and causing chaos in the kernel and before a new dinosaur is allowed into the park the virier um makes sure that the enclosure is safe also so it checks the basic layout in the first step to make sure there are no hidden tunnels or blind spots so unreachable instructions or Loops in the code and then like Park security that goes and checks the cage and you know Wiggles the lock and verifies and simulates all the possible behaviors of the dinosaur um you ensure that um the dinosaur doesn't uh try and headbut the

fence or like execute an

instruction um um right so yeah the first part is like the electric fence and the second part is like the Rangers in Jurassic Park terms uh so in the wild there have been three sort of high-profile cves associated with um using ebf as a means of breaking out of containers um one of them was this closed like quite recently by Google last month and we'll be looking at an older one though and um they all have one thing in common though which is the the main thing that made them work was maths so um it works because um um because of this arithmetic Lo logic in the ebpf verifier and it works because only the

true conditional path is analyzed by the verifier and we can put whatever instructions we want to in the false path and the verifier will accepted and the instructions in the de PA path will then be patched and um you can manipulate the counter in there and cause an infinite Loop and we can just use denial of service in there by launching lots of instances of that exploit and then locking up all available kernel threads and yeah that then in the end messes with the register and we achieve arbitrary right and from there we can then just go look at the threats task struct address and find the cred strut and set the ID to zero and voila we have

root um so the analogy here is that there's a blind spot in the fence where the cameras can see and um yeah the Raptors can escape if they hit the fence just right all right we'll have a little demo now um see if this will work

[Music]

[Music]

[Music]

give me one second

[Music] ch [Music]

[Music] all right I'll just reboot dra pack

[Music]

all right sorry about that so I going it a little bit bigger can you see the oh why is it black [Music]

[Music]

all right there we go sorry the demog gods were not with [Music] me um all right so um we have a simple Docker file here so this is a Ubuntu system or Ubuntu VM and also a ubun to container we add just a a regular demo user and we switch to that user and and we build the exploit after and um I'll add another tab to show you the running processes and and a tab to run [Music] um going to run BPF Trace which is allows us to uh see the CIS calls made by the ebpf program and I'll run the docker

container okay so if we check now so we're the demo user and by the end we should be root and just run the exploit here and in the meantime we can take a look at the processes so [Music]

right that's not what I wanted to show sorry I like messed up my screen well in the background it's um messing around with the arithmetic stuff that I mentioned it's making uh all those CIS calls and eventually it'll exhaust all the uh available kernel threads and let's get run

here and takes a little moment

all right so now if we run who am I we root and we uh we're in the root user group and um we can do now for example it's look at mounts and we can actually Mount the host file system um and yeah we can use that to write files to the host and uh other fun things like that well um so yeah that's uh we we got root on the machine so what now so there are a bunch of fun things we can do with those permissions so we can go explore the network we can do data exfiltration to Steel credentials and files um we can do lateral movements so we can spawn additional privileged containers if we

want to to run more things and be um undetected and of course establish persistence and install a back door can uh spread ransomwares in the in the systems and if it's like a mail server for example we can use it to uh run a fishing campaign or things like that uh um so yeah exploring the network is like um a dinosaur looking for weak spots in the fences and uh the data exfiltration you know downloading sensitive info like the dino feeding schedules or the electric fence codes and lateral movements is like the Raptors once they're loose they move from one enclosure to the next and yeah um establishing P persistence or installing a back door it's like the

hidden nest in the visitor center and Jurassic Park and then for ransomware and fishing I don't have examples because they're kind of self-explanatory um so what can we do to prevent things like that um maybe I have to say that um I I demoed it on yunu because it never worked in real and um that is because there are a lot of um access control things um baked into the system so like SE Linux the security enhanced Linux that has mandatory Access Control to prevent um run C from being overwritten so run C is a container run time and um also if you did get root on a system and you tried to disable SE

Linux um you'd have to reboot the system in any case so it um you definitely notice it wouldn't be like a stealth attack um there's also app armor which is similar to SC Linux in the sense that it has this mandatory Access Control model but instead of being policy based where you write policies um it's path based and um then there's also secure comp Computing mode which is a kernel feature that restricts the system calls a process can make and yeah it's meant to ensure that if a malicious program gets inside a container it can't easily uh make a Jailbreak by exploiting these system

calls so how does an exploit like that even end up in our containers um the most common way is like a supply chain attack and and there are a lot of different attack vectors um in there so you can tamper with the source code you can temper with the container file um you can attack the machine that actually runs the build for the container um using vulnerable base images is a big thing um if you're using public Registries it's quite common that people sort of typo squat so if you mistype YouTu or something you might think you're downloading a Ubuntu image but it's actually a vulnerable image that just looks like Ubuntu um and of course dependencies

that everything you package into a container um has the potential of having vulnerabilities or cves in it that people can exploit and then finally the registry the stored images can be tampered with and of course then when it's deployed you can also temper with that so the sort of proposed solution to that um which is like more or less the state-ofthe-art I guess um is using trusted software Supply Chain Solutions so redart has a lot of open- source projects going on in the space at the moment so some of them are meant to um help at the code level some at the build level um most of use cryptography in some way to um sign the images and then

you can also verify at runtime that what you're running is actually a trusted image and um with regards to like vulnerabilities there's um maybe you've heard of es bombs the um software bill of materials so you can actually have a really detailed breakdown of um the actual things you put into your container images and what is vulnerable what needs to be patched and it's a lot more actionable and the whole approach is really based on uh shift left methodology so catching vulnerabilities and issues as early as possible so you don't have to go fix things further down the line when it's more expensive and um takes more time to actually fix all right if you're interested in

the topic I have some suggested further reading so um these three books are really valuable I think and there are a lot of blog posts also around that space and yeah if you have questions for me I'll be around um at the conference and you can also reach me at Jos RH hat.com and yeah I think we can open it up to questions

thanks very much got any questions [Music] good um thank you very much for presentation um as edpf now we start be Ted in all sorts of products by surveillance politics G I know of and other companies do you think we're going to start see more and more issues with Cod getting into the Cur and sort of like um little verier have to be really hardened or improved to stop some of these potential esap going yeah definitely so yeah adoption is definitely increasing in a lot of products um personally I'm kind of split on whether it makes sense in most of them to even use ebpf over like some other Frameworks um but yeah the the verifier is kind of um

the biggest weak point I think at the moment and the the exploits are fairly complex and they're also complex to fix in many [Music] cases yeah thank you for the um lect I just wanted to ask you that demo you showed us it was on Linux would you be able to take on your admin on a server was based on Windows or anything else um that still apply uh not with this particular exploit but it's a good question because ebpf is now also like semi supported on Windows um but this exploit was um particular for Linux and is yeah

[Music] hello um do the exploit techniques significantly differ between um different container engines such as pman and do um yes that is actually an excellent question because I tried to get my exploit working with podman first and it didn't work so yeah the the main difference probably is around how it it handles permissions and especially with pman being more rootless by

Design okay well um as Adam said yesterday I'm not going to stop everybody from getting to lunch as well um so uh should we give joy another big round of applause