BSidesATL 2020 - Detect: Compromising Containers and Clusters

it is one o'clock on the eastern time zone and welcome back to the afternoon tracks here at besides Atlanta 2020 my name is Andy green a faculty member at Kennesaw State University outside of Atlanta where I teach information security courses and I'm also I'm also in charge of they screwed up and put me in charge of the degree program and information security and assurance so there's that what I want to do now is just kind of go through everything that I've been doing all day long and then remind you about a few things and then we will start with our one o'clock talk so once again I'm gonna put this slide up and for those of

you who have been here for a while you know how this goes but we literally would not be able to do this without our without our our wonderful sponsors and so I'm going to put this slide deck up once again and just to remind you that these folks all stayed with us even as we made the switch from virtual or from physical to virtual and so we'll start that we'll start at the top at the diamond level Warner Media thank you so much my bosses Kennesaw State University and écoles college of business where my Dean and her and her staff of folks have been really good to work with and my my individual department that I teach out

of the Kennesaw State University Department of Information Systems furthermore we've got Bishop Fox we've got coal fire genuine parts and NCR all at the all at the gold level moving down to the crystal level we have a critical path and synopsis coming into the Silver level Aran's we also have binary defense Black Hills information security core light and guide point security coming in at bronze we have the NCC group which we just heard a bit about and Rory is with the NCC group and so we thank them for their sponsorship of today's event also a big thank you to our in-kind sponsors yesterday E Council came through with some with a pace training opportunity that I think most of you or some of you

were able to take advantage of hopefully you found that beneficial also secure code warrior has been running an online CTF for us all day long and it's still over there ongoing right now as far as I know next we'd like to thank a few individuals and organizations for contributing to our raffle prize effort might cost a crosshair information-technology Joe gray offensive security and the pen tester lab also I want to mention that if you have not yet done so this is a virtual global conference and we are curious as to where you are coming from so I'm going to paste this link in the channel and I want you if you haven't visited it already I would like for you

to go drop a pin and let us know where you're from and take a look at the map and see where folks are checking in from assuming that everybody is being honest we've got folks in from Australia Germany the EU and of course a ton of folks here in the United States so if you haven't dropped a pin yet please go do that so that we can kind of get an idea about just how global our reach is at over 1,100 attendees today it's it's it's a sight to see I also want to remind you about the raffle prize giveaway that's going on if you have not jumped in to this channel that I'm about

to drop [Music] right there in the channel if you have not jumped in the raffles and giveaways channel to get an opportunity to win some of the cool stuff that sponsors have given us to give away go ahead and jump in there now I have done enough talking and so now I'm gonna ask you to please give a warm welcome to Rory Mattoon with NCC group who will be talking with us about compromising containers and clusters and I will stop sharing my screen now Rory so you can go to or so um the goal of the talk today is to talk about compromising containers and clusters water warriors talk a bit about it was looking at some common

container solutions I'm guessing more and more people will be coming across the taking care of an attackers eye view point and looking at how an attacker might try and compromise maybe docker systems or kubernetes systems and this is something that you know we do or I do quite a bit hopefully got some useful information in there for you before I get started on the top a little bit of an about me I've been in information and IT security for about 19 years before that I was in ite as a network consultant who remembers that I'm a principal consultant with NCC Brook in the UK these days I spend most of my time working on containerization that

have web app stuff from but cloud but mostly containers I'm a career up Rory I'm sorry to interrupt just I just thought about this tell folks where you're from and what time it is and and how you came to be here today so yeah I was doing this talk on Wednesday because unfortunately one of our other consultants who have lined up I was unable to to make it they had anything here to do so I stepped in and as I like talking about containers this was just another opportunity for me to do that so I'm as you can tell from the accent and I'll show you a picture in a second I'm in the Highlands of Scotland at the

moment in a little village called loss coil head so if there's any audio it may be either it's traveling a long way which might be wise that up so yeah Alison I'm a contributor at security Stack Exchange which I always mention in because people may not have heard of it it's like a smaller I'm friendly or a person all stackoverflow with a focus on security so if you have any security questions then it's a cool place to go and ask them or indeed if you want to answer a few questions and the last one which is kind of relevant to this talk is what I got into containers here do you like sort of three or four years ago

now there wasn't a lot of documentation so I kind of helped I'm on the CIA's benchmark for doctrine curiosities so if you're looking for information about that then hope you'll get it the picture is just coming hopefully see the picture man I just didn't think was in there wasn't a picture there's about to be a picture there's the picture I took this this week this is just the village I stand I just went Bob the village so hopefully everyone can see that here let me let me let me poke in here I am not seeing your screen right now yeah I know just the green box went away sorry I had to take over - yeah yeah we

can see it now I'll ship you a beer I owe you my bad awesome so yeah that's the picture and that coil here in this house hopefully you can just see the one spirit easy bill yeah cool so that's where I know and that's just above my just above where I kind of go for a walk a bit of a morning so anyway Oh what are we going to talk about today we're going to talk about is we're going to talk a bit about what is docker right before you look at securing or hacking into or whatever you're doing with the technology it's a good idea understand how it works so we're to start there then we're going to talk

about attacking docker locally so say I'm a pen tester and I've got local actors to the hosts but I'm a North Lee user and I'd like to privilege escalate right that's a standard kind of pen test attack something to do if I get access to docker how do I do that attacking docker remotely sometimes people will make docker available over the network and that can be bad so we can look at how that works then we'll talk a little bit about what Canaries is if you've been involved in containerization a lot over the last couple of years you will probably have come across communities these days is pretty much everywhere and it's worth talking a little bit about

how it works and what it does again understanding these things helps us when we want to secure the more we want to break into them I'm being talked about attacking your Bernanke's remotely so we can talk about how we do this if you find this of a network pen test and we see like whatever it is cluster what can we maybe do to actually kind of break into that cluster and actually connect compromised bits of it and then what we'll do after that is talk a bit about attacking kubernetes locally so Cuba Nettie's is a multi-user system you know when I'm reviewing a cluster for somebody I might see a cluster with maybe 50 or 60 applications deployed

onto it so the groups of developers you've got access to that cluster and obviously we don't want to give every developer full control of the cluster so we want to try and say okay if I was an attack prep say I'm a pen tester I've got access as a single developer can I escalate that access and can I essentially control the entire cluster we'll talk about how you can do that as well so first off is to talk about docker and a little bit about where it came from and how it works which i think is kind of useful for understanding what we'll kind of talk through the rest of it of the fee of the talk so talkers

bit of an overloaded term and there's a number of different ways it gets used like everything in technology a lot seems to be like a number of different you know things that are referring to the same term there is docker the company right so darker ink is that company and then they there is a product called docker right so they have a product which is called docker there is also the open source project which was called docker book isn't there called movie which i think is for trademark reasons or something like that they renamed it so if you go to github and try to look for docker you just get me directed to mobi there are two products

there's docker C which is the community edition this is the vast majority of what you'll see if you're actually installing or using docker this is the open source version of ever they can freakin own commercial version there's also docker ee which is enterprise edition which has better support and some additional features to confuse things slightly dr ee and the enterprise version got sold to another company called mer antis in november last year so i just even more more confusingly but for the purposes of this talk water table is dr. Steve so we're just gonna talk about the open source project we're not going to talk about the commercial stuff really at all so when we're

talking about containers first thing to understand is that they're not virtual machines and they're all really like virtual machines what you'll see later on is when we do the demos they came to look a lot like virtual machines so let's just talk a little bit about that typically when you run a virtual machine and you are an application on it maybe an ec2 or on-prem and vSphere or something about you have an innocence to the Linux kernel and you have a single application which runs on top of that Linux kernel matter of vm there with containers what you do is you say okay I got this VM I've got a single instance of the Linux kernel and then on top of

that I'm gonna put a number of containers and each one of those containers is going to have an application running in it now this probably is why a lot of companies let's definitely what I see from companies we've talked to a lot of companies like containers because if I'm running easy to for example I pay per hour that my ec2 instance is running if it's 1% of your flies i peter semi-matte money it is 100% utilized so if I can put more applications onto my ec2 instance I'm saving myself money this is why I think a lot of companies like containers because they give you a way of doing that kind of good easy reasonably

successful way of doing that you can be run several applications where you run one so you can cut your ec2 builder so that's kind of an important point to note so what is a container well it's not a virtual machine that's a lot like one but it isn't one literally what a container is this is an important point or is danica tiers is just the Linux process right it's literally just a process like any other process on a Linux host except it's isolated and darker layers various existing limits technologies things have been around sometimes for in well over 10 years now and to create this kind of isolated environment for each one of our processes so they can't mess with the

other processes or the underlying host dogma containerization started with Linux so it was originally a Linux taking but you can now get doc or Windows and we're starting to see some of our customers on the tests I've done over this year last year our doctor on Windows Microsoft could be very large amount of work into making containerization docker style containerization work with Windows so that's available now if you're using wet women is Sarah 2019 or above you can get windows containers and again it's a single kernel per host I put a little star next to that because there are some ways of running containers that actually although it looks like a container it's actually a virtual machine which that's

can like go yeah it's a bit confusing but for the purposes of this talk we'll stick to two standard Linux containers and that Carol is shared between each one of the container running on that host another thing to mention about this feeling is kind of important I definitely annoys a lot of kind of people who've been in the UNIX world for a long time it's opera has all this attention now and they say well this isn't you and they're completely correct containerization started in about 1979 and with the children system calling unix so very much on you then around between like 2000 and 2004 freebsd linux and solaris some level of containerization if 2008 a protocol

Alexi came along which is still very active project in the container space and then 2013 doctor came along which was based originally or LHC but then and then did went over to his own kind of technology stack so this isn't new but I'm guessing probably people have realized that they were hearing a lot more about it in the last couple of years and that's because of dr. central and the reason for that is it's easy it's convenient it works well developers like it because they can get their applications they can package them up and say the docker container and they can have the same container running on their laptop and then put it into a CSV

pipeline and take it into a test environment and run the exact same container there and it generally works and then take that exact same container and put it in production and it works there too right this is very very handy this is something that I think that's why developers attracted to businesses sips of money developers it's a nice easy way of doing things typically whenever we CC ICD environments so things like Jenkins setups will see containers because it works very well with that kind of automation doctors providing isolation for our processes and then obviously you could just want all your Linux processes together on a single host but they tend to get in the way of each other things like library

version flashes happen packages have conflicting versions and doctrine gets rid of all of that kind of problem so they provide a level of isolation both for kind of practical purposes also for security in doing so though they provide a lot better and resource utilization than VMs so a virtual machine image might be hundreds of megabytes in size you can get a valid container image in like five to ten megabytes so you're talking much smaller and much better resource utilization so that's the killer VIP view right that's the IT view of what this is let's talk about the pen test review the attacker view docker is command execution as a service that's what it does right so when you run docker what

you're doing is executing commands on a host come on diffusion as a service if you put docker on the network and you make it available over the network its remote command execution as a service that was a pen test without something really effective I got a thing that if I can access it will allow me to execute commands on a host that's exactly what I pentester I tend to want and one of the good things from an attackers perspective is docker has a flexible security model it's defaults aren't terrible but it's designed to have a very flexible so you can add and remove parts of the security model and that flexibility in the hands of an attacker

can make it very useful with emulation run containers and maybe break out to the underlying system and a way there that you wouldn't necessarily expect as possible so Dockers flexibility is kind of useful for attackers so let's talk about and make this a bit more practical so we talked a bit about the theory there's docker wax of how it's set up let's talk about how we would actually attack it let's talk about some local attacks so for these attacks we're talking about people who got access I neither to a host running docker but maybe they're not pushed and only an ordinary user or maybe they're gonna access to a running container and what they're trying to do is break out of

that container and get current leaf access on the underlying host who'd we go around doing that first point to note in a very important point for security is the dr. Damon runs as root right so the dr. Damon because it does a lot of low-level things like modifying formal rules inventing file systems and all that sort of stuff it runs as route and as a result it tends to be you know if you can do something banner that you can get to you root if you can run docker commands it's likely you can get rid and we'll demonstrate how that works it's possible to configure there so that that's not an issue but I honestly have

hardly ever seen a client configured opera in that way so usually if I can run docker commands I'd be real the host is just a question of which command do we want to run and important point to know from security architecture standpoint is this is by designing docker do not regard they don't have an authorization model just with docker itself it's designed to be AI if you can run it you have access to access to the hosts so whenever I'm talking to a customer and I say how are they configuring who's your the access to it I say assume that anyone you were going to give docker command access to is root just like design your system with that

assumption in mind because that is how it works that's that it's by design it's not like some super-secret concept this is how it's meant to work so let's do a demo looks can people see my console okay if that's working yeah awesome thank you so I am Rory I'm I'm not root access to your foot and it'll say here gig here at your daughter y'all good so I have got a level of access to the source but I'm not rude if I'm a pen tester at this point what I want to be is root what do I do this I'll explain how this command works in a second but let's see what happens if I enter oh

look I'm rude there we go and the root user I'm the real root user that's it right so that's kind of when I say if someone has access to run docker commands their route it's because of that you've just rude so let's exit back here we'll go back I think I just did stop what that was the command I just ran there is what was originally called the most pointless daughter command ever and this was written in a blog post I'm at 2015 by a chap called in meal and I call it the most useful command ever because I use this on tests all the time well this basically does is it says okay I wonder on top come on so then I run so

docker run this just says give me an interactive shell insight that the docker container I'm going to run this bit here which we'll talk about about more later on this is take away all the security so you remember I talked about - - privilege I talk about privileged and doctor having a flexible model this essentially is that flexibility in action so this is a take away all the security usually in docker you don't get you get your own private networking stack so this is now I don't want that I'd like the hosts networking I'd like the hosts process list and then I'd like to meddle entire fastest turn from the host insider directory and share it to that directory effective

alike is basically give me a root shell on the machine that's literally Matt that's yeah so that if you ever ever ever says to you you know I've got dog praxis does not mean up but root the answer is yes just learn that what really will so that's the first thing if you have to or accessing your fan tester from that you get to be really the second thing to note is that the way that docker works is the command line claim you're running there talks to the docker demon over a socket pal so there's actually there is a docker socket and a socket if you haven't come across there's just a special linux file

that lets you talk to a server without opening a network port so as you talk to a file you saved it commands the reason I mentioned this is a lot of docker images will suggest mounting the docker socket inside the container and that's a dangerous thing to do I've seen this quite a lot in monitoring tools in it management tools so anything that says it wants to like monitor docker or manage docker will say he just made the the darker socket inside inside the container and not be great this is effectively giving my third party software root on the host so if you get one or two tool when you do that make very sure you trust someone during tool

because it will give them real on the hosts so we can demonstrate that what I'm gonna do here I'm gonna run another dollar command but I'm not running any of that stuff I did before with privilege right so there's no - - privileged here there is no and removing the layers of isolation the docker proper provides all I'm doing in this command here is I'm saying move the docker socket so this special socket pal inside the container at the same location so mount a foreign dollars off and then we're gonna run this container image it's just just one I have a lot of tools loaded into and we'll run a bash shell so when we do that we get dropped

into a car shell notably I do I'm isolated here right this is why docker containers look a bit like virtual machines right cuz I've got my own IP address I've got my own process list and I've got my own file system but I've also got access to the docker socket because it was meant to didn't set the container so once that's happened we just okay our most useful front ever run it and ruin the host again right so here I am I'm back on the host and I'm gonna and the full root user so if I do yes yeah for something and the full root user so that again if you give software access to the docker socket or if you're

a pen tester you're doing a test and you encounter you got yourself into a container and you have dropper sock access then you get to root that's how it works um this is also kind of useful for CTF sometimes CTF people will go to the dock or socket in as a kind of a trip so you land in a container and it's how you pray okay and that's how you break it you just run that come on so I'll get back to my host again so so that's doctor socket I'd say the other one to mention the other one to talk about with docker containers is this - - privileged flag this is something which doctor came

up with and I don't really I wish it wasn't there as a security person but as a fan tester it's kind of cool a lot of container images will suggest giving their programs they what run as I container - one is privileged it's a really bad idea essentially this is remove all of docker security isolation if you do that you're saying she's saying give whatever is running inside this container access to the underlying host generally as route so don't do that but it's something we do see and there are the odd occasion where it's justified but the majority of time I I think if it kind of like in a Windows domain giving someone to me an

admin because they need some extra rights you say oh I don't know exactly what you need I was given to me that kinda like that it's it's it's a bit of a lazy way of doing it so it's not a great idea well it's uh it's definitely something which we see sometimes on tests so if we've got privileged container we can break out of it using this which is cool because I like exploits that fit inside tweets and this exploit fit it inside a tweet from a chap called Felix

which our Linux feature to allow you from inside a polish container to execute commands on the underlying host so if I'm in a container and it's privileged to have a pen tester you get a copy of this script get it inside the container and you can go from there so we can demonstrate that to demonstrate so another Noah run container is we're doing - mine is privileged but I'm not doing the docker sake and I am NOT I'm not meant to give any other file systems I'm just going to do - - privileged yeah you're you're chirping in and out a little bit on your audio roars push on and all hopefully it's been fine for the

most part that I just got a little chirp yeah less about your back you later I mean I've learned a lot nice monster it shouldn't be bad with I will say that you'll avoid live in the middle of nowhere I actually have decent internet I have full fiber so I'm gonna be really disappointed it is that so anyway let's let's let's hopefully it'll stay happy and it'll do its stuff so we're now inside a container again we only have the process list for inside the container and we only have the IP address right so we're a restricted we're inside a container but we've got this - manners privileged so we can use our fancy script to actually try and

break out so I thought the script meant and I'm one if I were on that's the script essentially that's the script page or the slide and we're gonna run this convict now we're on that what we see is the information the underlying host so that's my underlying host IP address and everything else so if you ever land that a privilege container you just run this escape shred script and then where you go we can also do something like on your blog post and the answer is ever so privileged containers are a super bad idea super dangerous and if you're a pen test are fairly easy to break out off you essentially just get that the scripture

of the tweet and and you can run commands quite happily as route I let's keep a sure it's just this look at that I just I just put that into a into a file so that's just all this escape of shows I can actually show you it yeah so it's literally just the justice cryptic erratic this is our lexer I mean this container image here this is just one of my doctor hot beverages and on docker hub you can just get that which is mine obviously one of the important things I don't we talk about too much on this talk is don't pull random images from dog or hub you don't trust because they could have literally anything in them

but if you want that's on github as well so you can seal the toolbar installed so or a pre-release containers so other attacks as well yeah there it is it's a really cool tree it's a really cool hack other ways we typically see with docker containers on their own people breaking out in appropriate volume mounts so maybe not mounting the entire post file system but the mounting something like etc' inside a container is not totally unheard of that's an avenue to look for and kernel vulnerabilities so this shared kernel is probably the major difference between a docker container and its default setup a virtual machine if people are running out of date kernels and I think one of

the speakers before was saying that let him this morning was saying that there one of the things they noticed was the container host don't get me booted or don't get patched very often that's that's true we see that I have seen things like your vanities clusters where they have an uptime of over a year which he tested me you're not touching the kernel of this host otherwise it wouldn't have the uptime of over a year so kernel vulnerabilities as well are another one for making outdoor containers although if you keep your kernel up today then unless your attackers are packing O'Day's you probably don't have too much to worry about there so don't promote attacks if you are actually do

over the network so by default docker doesn't listen remotely so by default docker is actually quite good it doesn't have an attack surface on the network it only listens on the socket file which is var run docker dot sock however it is possible to configure it to listen remotely the two ports to watch for 2375 TCP is the default unencrypted port and at this port has no authentication and authorization by default it's theoretically possible to add it but I any time I see that port it's not it's it's this is literally remote code execution as a service if you see that poor you probably gonna be able to basically execute code of the host 2376

is the default encrypted port that generally has client certificate authentication so it's typically authenticated but but this is the one that there is a of serious concern if you see it and we do see it this isn't like a theoretical thing we see this on assessments reasonably regularly so the way the doctor works the way that the docker demon works is it's a REST API alright so the docker daemon is literally just a an HTTP web service style REST API you send it any valid HTTP command and it will act on it it's generally on earth educated under cryptid as I said because it's a REST API it's literally just you know a HTTP thing we don't need any fancy tools you

leaked hacking tools we can use curl so we can become hack systems with curl so let's let's let's see if we can do that so what I'm going to do is on our host so on this host I have got the daughter daemon listening remotely one thing actually to know is even even cycling servers where they will configure it list and remotely indeed there was one on a job recently I think last week I've also seen a lot of development tools will recommend that developers make doctor listen on localhost to 375 which isn't obviously as dangerous but still isn't a great idea so it's worth noting couple of ways that could happen so how would we hack this with pearl

right we're gonna try practicing this thing and actually hack mr. dinner and actually hack the host with just curl and SSH the first thing we need to do is we need to create a an image make sure the image is on the docker setup on a given host and the so we've got a curl command that looks a bit like this here's our 2375 port we're going to see images create so create me a new image from the dropper hub image our pint container tools so let's just play the image that used before and when we do that it says sure

okay lovely what we can do is we can say okay I want to create a container from that image so you see here this is just again it's quite kind of standard resti yeah the end point is just containers create which makes sense right so we've gotta we want to start a new container based on that image but notably what we're gonna see here is give me a privileged container so from remote we can just say hey give me a privileged container just like we did before when you find out that privileged essentially gives you a route on the host you can do it remotely so it says yep sure and it comes back with this in the response what we're

seeing here is we get a sha-256 hash docker everything doctor does images networks containers it identifies them with sha hashes i generally give you a friendly named work with as well but but the shark hash is the real name so we need this because this is now our containers name and we want to do anything with it we have to we have to use this so for the next thing what we need to do is we need to actually start that kikuna so we've got it on them to start up so ii command the command me issue so just looks a bit like this we say get the container called this which is just the name of the container and

start it and this shirt so we've moved got a running container so we've created a privileged container and a host remotely this could be our other Network anything you can get to on to 375 and we can create a privileged container privilege frost essentially running on that host and this is because to Iran SSH inside it so there's an SSH daemon running inside my container important point to know about containers is usually whilst they might run one process there's nothing to stop you running as many processes are you like inside the container so you can run SSH inside the container it's not generally considered a good idea but it's good for fantastic so once I've got her I can do this which

is a look at the logs of my kid so my container is set up I need to actually give it the idea what this container does is it actually sets up a new root password so obviously I don't want to put my root password inside my docker image because if you hard could root password and your program and just people both steal them from you so what I've done is I've got I know got a host I continue the running on that host with the IP address and with that root password and I can SSH to it so I do if I'm running a stage or a high port so it doesn't reflect well when it'll say

you've got my hosts so let me know so I can connect to that give it my password and I mean the container on the hood I can be done over any network all right so you can do this remotely and now I'm in a privileged container and so now I've got my escape route

and number so I'm Groot on the underlying host so if you get darker as your pentester and you get four 2375 even if you only have pearl you can basically get the point of compromising the host because docker is remote command execution is a service which is exactly as a pen tester what we like to see so obviously no one would do that right this would never happen normally they ever do this in the real world this is a showdown search I did two days ago when I was put together the slides for this at the moment there are five thousand seven hundred and seventy instances of what is pretty definitely docker running unencrypted RCE as a

service these are pretty much awkward to be running cryptic way meaning people who like cryptic remaining have worked out that docker is remote code execution as a service and you can download manila miners on to hosts and use all people CPU to made wherever coin you're currently mining this number goes up and down actually five cents it's quite a lot higher that used to be used for three days more people doing this people do make this mistake it definitely happens so that was docker and I was a kind of reasonably quick went through so the doc Griselda is a fairly simple product it doesn't have you if you get access to your right that's just the way

it is it's not really designed for anything else however docker is designed to run containers on one host so where you're doing dog where you're running containers of a single machine however if I'm a business I want to run containers on lots of machines so I want to run clusters of ten one hundred you know thousand VMs and I want to move containers around those easily and I want to deploy sets of containers onto those hosts easily and that's where accurate ease comes in it's designed for that kind of scenario Nettie's is container clustering and orchestration so it's literally taking these processes these containers and having sex of them deployed across the set of VMs it was started by some people

within Google it was more of an internal product the Google have got called board and it's visibly implementation of those ideas these days is open source or unmanaged or I the cloud native computing foundation so CN CF if you haven't come across them they are a subset of the Linux Foundation and they manage and cloud native projects they've got like a wide range of projects that they essentially kind of help to foster adoption of an unmarked and all that good stuff one important thing that we find that trips companies up when they're deploying kubernetes is that it's still quite rapidly developing there's one release every three months or so the last release was I think three

days ago and the only support the latest three releases what that means in practice is if you're running humanities you have to upgrade your clusters every one of you nodes in your clusters at least every nine months so otherwise you're at support and if there's a security vulnerability which like anything there are always security vulnerabilities your app support and you won't get patches there are no long-term support releases really of cuber Nettie's and there's one or two options but not many so what no did not my test trippy blob of it one thing with kubernetes and it takes a beginning used to is there's lots and lots of different ways of deploying it there are is a spreadsheet contains

about 100 plus different ways of deploying kubernetes there's managed options so any of your cloud providers which are a cloud for any of you with will have a managed cuban at ease so google containers and gke is google's then there's all complaining if you're in the Red Hat world IKS and dks so basically whatever cloud you have will probably have a managed German at ease with these products they tends to you have less to worry about you don't have as many things to to manage they do better for you on the other hand you have less control over your environment you then get like an on-site platform as a service something like open chef container

platform gke on-prem they'll provide things like management gooeys and also kind of tooling for you or you can install urban eighties with just terraform this products like put is like corpse QB the m because if there's literally one of the problems of curiosity securities if someone says we'll what's this you know if this is a good value or a bad value and it depends on the deployer i depends on which one of these different platforms you're using how does it work what this is kind of a this is the most basic humanities cluster set up the weak humanities works is it's already container prices all around these daughter containers and deploying them kubernetes has in the middle here we've

got this api server and the api server is just a rest api so it's a essentially an HTTP api that you send commands to to create and manage containers which are just processes the EPI server in the kubernetes cluster is absolutely the heart of it like all the other component you see if the way the diagram flows everything talks to the api server and all the other components it'll talk directly to each other they talk to api server so this is why it's encoded in red because if i can attack her this is my obviously my number one target if i compromise the api server i can compromise the whole cluster at the top we have a thing called a CD so

kubernetes is stateless by default the api server doesn't store any state it handles it all to this as if he is a key value store so a kind of a simple database it's not like a complex thing like oracle it's a fairly basic key value store but that's where all the state of my cluster stores all the information about what workloads are here what things have been deployed to it as we'll see later on any secrets that are stored in the cluster they'll live in here so again as an attacker as a pen tester I want to target this because this is all the cool information and then in addition to these master nodes which is in if they

have become a via the control plane there's a lot of worker nodes and these were my containers actually go and then here we have this thing called the cubelet and the cubelet essentially is a thing that sits between the api server and dark this boxes container engine doesn't have to be docker but every question I ever review is docker so all of what this does this sits on top of watercress so doctor says here in this corner doing what we just showed it running processes and the qubit tells doctor what to do so if I say give me a new web server give me five of them I see that today if five went out first it then says okay

cubelet give me one of those web servers and the culet sister docker give me one of those web servers and dr. justice docker run all right docker run just like come on to round before so obviously if I can control humor then I can control docker and I'm ruling the hosts if I can control the API server I can control the Kibler which controls docker which makes me a rule on the host and xcp will see has some interesting information for that as well so from an attackers perspective as a pen tester if docker is RCE as a service kubernetes is distributed RCE as a service literally for the thousand nodes in a cluster if I

could control the cluster I can execute commands on every single one those nodes as root so it's distributed RCE which is very nice it has a lot of network attack surface so docker as we saw earlier on doesn't have a lot of network attack surface there's not a lot to get to with kubernetes there is and there's a lot of complexity of securities unlike docker which is reasonably straightforward once you get used to it Trinity's ain't gonna lie to you it's just a bit complicated so network attack surface if you're a pen tester the first thing you ever do is look for ports right you're a network pen test your scanning host you're looking for all the

services this is [ __ ] with kubernetes api server is the eating it can be on a number of ports these are the ones usually you've c64 for fleas believe most common 443 is pretty common as well the cubelet as a 410 250 which is which is its port forwarding on and sed 2379 does the other couple is also tend to be a little port like metrics and stuff so typically I wouldn't worry too much about these they're useful because if you see those that tell you hey this is this is a cluster right I see ten to five six and ten to five - it's pretty much going to be accurate he's prosper but these are

the ones were going to focus on because they're the most interesting so let's talk about attacking how would I attack this right I want to actually attack these clusters I want to see that compromised them we'll talk about the three main components so the thoughts places to go unlike docker there's lots of things we can try the key words to consider a PR server Etsy D and Q so let's look at this what to do let's do some demos cuz they're working so far okay which I'm with my audio the demos seem to be going okay how is the audio whether is it still working we're still with us audio has been good for the most part just a

little bit of random clipping here and there awesome cool so what we'll do for this at Dell I'm gonna get a little client machine so this is a just a sage little client that's got access to our cluster we've got a couple of Q&A T's clusters running here that we can access so the first thing to do is if I get the EPI server then and if there's been a mistake made and authentication has been turned off then I can just execute commands so this command here uses a tool called cue cuddle our cue cuddle or cube CTL no one really knows how you pronounce it and this a sense is what you use to

manage given at these clusters it's just like it's gonna come online kinda like that don't go come on Blaine you use it to manage clusters we can tell it to connect to our cluster so I'm a pen tester I can just point this at any machine that's got this port open and try some commands and then this came out here essentially is going to execute a Linux command inside a container in this case it's an API server container so the candy API server itself actually runs as a container in the system I want to do is we're going to ask it to to give us the contents of a while so if we run it

we get that back which is an RSA private key now you see whether that war well if something is mistaken allowing you to hit the API server without authentication and this again this is not known we have definitely seen their self-interests people do make this mistake and people make the mistake on the Internet as well I think Tesla had at least one issue where they had to work their clusters online that was come from us this way um he serves his command execution as a service right so it allows me to execute any command inside any container running in that cluster now the reason I did this particular command is if you're a pen tester this is the golden key this is

like you know you cannot get an active directory you get that kind of a attacks little looper system access this is essentially the kubernetes equivalent the way that kubernetes works is it has client certificate authentication turned on by default on every cluster and there's no real good way of turning it off as part of the functionality of the product and the important point about the client certificate authentication that kubernetes does is you can't revoke client certificates so if someone gets a client certificate which gives them access to the cluster there is no way to revoke that certificate short of invalidating the entire certificate authority so as an attacker the thing I want is this client miss CA key file

because if I get the CA key file I can create new users users that will last as long as that certificate authority is valid which is generally measured in years so as a pen tester if you're looking at your registers that's the thing you want to get to on most clusters it's in that directory what work on every cluster as I said there's many different ways of installing this so this won't always work but if it does work that's the follow you're looking for so here all I've done is I've said hey go to my cluster execute the command and get this done and I thought works that's what you get so that's the API server we've

got the cute look so this Kuebler as I said the pupil it controls dock around so anything like the doctor can do if I can control the Kuebler I can do as well the qubit is on just let just let the API Center so you can connect to port 10 to 50 and we can also get to this pod endpoint here and if you're ever doing stuff with JSON everything in a nice line is JSON JQ is a really cool tool you pipe your output through it and it makes it look nice so I can do this and this gives me in a run another host so like I said the cubelet manages the containers on the host so you can get

everything you need out of it and the cool thing you can do is shoot it with a giblet the interesting thing about termination is this this was this vulnerability by default in older versions of humanities there was no authentication on this board and took about nine months to fix the problem which wasn't great so for awhile this was like the most reliable way of compromising clutters you can hit the cube look and say hey I want you to run a command and I want you to run it in a specific container this is our API server container again and then you can just give it any Linux command here you want so again I'm just gonna say hey

count it without clown and it says yeah sure there we go so again I'm a pen tester you can access the port ten to fifty if you can get to the API server you just carry that far and then you've got persistent access to the cluster because you can just create your users I mean just to carry their mystery that's not so come on that does that I could do something like ya know this is literally just literally command executions of service if you get the couplet this isn't as common to see oh it complained

PSSs although happy well if it does have to have to climb in there otherwise it won't work anyway I can do in this case the one you really want is just that because it gives you the CIA key so masculine is the database a CD is this key value store and it essentially has everything about the cluster this again others indicated if you're connecting to it you can't use curl so unlike lots of other things with just HTTP api's HCD uses a thing called G RPC which is a binary protocol from Google so you do have to use again it's called XE d couple select cube couple there's a tool called XE teakettle and it essentially

lets you query xyg databases so we can do that we can say hey give me all the keys give me everything in the database and I just unauthenticated let you do

[Music]

we can exit on the right let me let me actually see how we do that with a CD what do you seek Romeo here yeah Rory yeah we're your your audios clipping in and out again I'm not sure if it's au issue or zoom issue or an inter webs issue in general yeah something being funny yeah we've got just a couple minutes left how do you how do you want to proceed

resume excusing CPN but everything else is fine yeah as you say exploded okay how we do for time let me just a couple of minutes left Rory before we have to move on cool so very quickly say you can dump her pool tokens with sed it looks like that so again sed another place to make sure you keep on on it and keep it secure so let's rip past that because they weren't that interested oh good so I'll call the slides up and I'll put some demos up as well the important thing to remember of activities and docker is this docker is remote command execution as a service by design if someone gets access to docker

if you're a pen tester and you get access to daughter you probably may drill on the host it's not designed to have a kind of granular system is our CEO service cube Annette is distributed command education as a service so humanities for a pen tester standpoint is like getting something which designed to RCE and lets you do it remotely and as you can see if you give access to those api's you can generally do some fairly bad stuff fairly quickly if you want more information a couple places to go I look with the slides up into the slides channel there's some cool people to post lots of cool people to follow on Twitter I had to pick ones that would fit on

this slide these are members of cig honk in cold water particularly and Brad Guzan and that's my handle so feel free to follow any of us on Twitter there's a playlist there's a lot of good videos a good content which goes into more about container security if it's something you're interested in and also there are slang age from before and hopefully yeah yeah I'm just about to time mark so Wes let's wrap that up there I don't think there's any questions do let me know that's excellent thank you very much Rory for the presentation