Securing and Pentesting Kubernetes

can you switch to my feed does it is it good hey all right it works

all right we are going to get started

so those of you that aren't here yet just let me know when you get here but we'll just we'll go ahead and continue from from that point on um welcome to securing kubernetes or what i like to call the great spaghetti monster the flying spaghetti monster uh or whatever friendly little name you have for it i'm kat fitzgerald um i have been doing this for a very long time how many people go back to defcon 3. okay i'm gonna leave now because i am very old um wow oh well um yeah i run b-side chicago i'm also part of the b-sides pittsburgh team woo-hoo so yeah i've been doing this stuff since defcon 3 which is a very very long time

ago google is a very uh is our premier sponsor for b-sides pittsburgh if you're looking i am hiring i have a very large team at google so i'm just throwing that out there come see me later i'm based out of seattle honey pots and refrigerators are something that i love to play with a lot especially when it comes to kubernetes because i use kubernetes for deploying some of my honey pots and i love to cook and the reason i put that up there is just last weekend when i was home i decided i wanted to put up a new spice rack in my in my kitchen so what do you think i don't know is it

pretty cool huh okay anyway uh my disclaimer the views and opinions here are mine and mine alone have nothing to do with my employer or current or past um and anything that i share from the standpoint of security vulnerabilities etc please always take them with a grain of salt do not go and commit crimes only do things to systems that you have control over and you have legal rights to do so so why are we not here today well i'm not going to solve all your kubernetes um security problems that's not going to happen in 45 minutes neither will the person and even though it has crossed out sitting that's because i never know if i'm going

to give this talk virtually or what it's going to be but it's so cool to see people again so welcome all of you back to besides pittsburgh and something that i like to say is is common sense kind of went out the window a long long time ago uh part of that is is or that is one of the problems that we have to deal with we'll talk about that so why are we really here well kubernetes for the most part people are still getting into it it's still kind of new the reason i say it's new is because it's always changing and everybody's still kind of figuring it out and you never know but it's fun

and there are very quick and dirty ways to get up to speed on kubernetes if you happen to be running ubuntu you can throw microcase on your system or even on a mac or pc you can have it up and running in a matter of five minutes or so so think about that containers are not new and this is part of the problem if i were to ask you right now which is more secure a virtualized system or a container based system what would you say don't answer that but think about that question okay and security is for everyone i like to say that because i like to get people interested in doing security and

i lied common sense is required so please disregard my previous slide okay um something to keep in mind when we talk about kubernetes most breaches are not zero day when you look at all the breaches that we see today they're not really fancy they're not caused by these these zero day attacks and so on most of the breaches don't come from ooh somebody ran a vulnerability scanner and figured out that that there was a problem and then they broke into me and and so on no most problems come from configuration issues and configuration issues when you start to think about kubernetes are huge and containers okay a close second happens to be compromised credentials i still

go on github you know once a week or so and i just search for keywords like password or root or aws or token or things like that see how many occurrences you will find and then the third trailing um compromise is over privileged accounts this is also important if you're going to be deploying kubernetes in the cloud i don't care which cloud you do it in well i do but no which cloud you do it in it's going to be a problem because a lot of people just go oh wait what are all these i am permissions what oh gosh you know i really need to do everything so i'm just going to give my account everything and

it's like root okay fine how many people still log in to root on their linux box and do all their work all day on route uh you're all lying kubernetes really has all three of these types of events that can cause incidents so we're going to look at that as we go by the way if you haven't figured it out yet yes i do like cat memes i don't know why containers we have to kind of start with containers before we get into kubernetes containers are not secure by default when i do interviews of potential candidates i typically will ask the question that i just gave you a few minutes ago is a virtualized system

or a container-based system more secure and several years ago i asked this question of a particular candidate and another company that i was working at and they swore up and down that if they took their application that had vulnerabilities and they moved it into a container that the container would fix all the vulnerabilities of the application and i kept trying to get them to explain it to me how it would do this they went through a very long explanation which ended up as c the application is now secure and i was just like huh because containers are not secure by default and they don't change if you have an insecure application so please keep that in mind but

when we work with containers we have resources think of a container as a grouping of these resources or a grouping of stuff stuff equals resources resources are named spaces if you do not apply certain controls on your namespaces you will have an insecure container okay namespaces will contain things like processes cpus network ports all these various things resources can be exploitable what if you have a container sitting on a system that has oh i don't know 20 cpus on it and and you know um a hundred gig of memory something like that but you don't put any limits on cpu or memory usage of a container and somebody either tries to do a denial of service

against that container the the application in the container or they actually break into it in some way well they bring down the whole bloody container and the system so always remember that resources are exploitable so first thing we're going to start is to look at how we secure the container we're going to do that with control groups control groups are limits so we want to apply limits to these control groups come on there we go so if we're looking at a container you know how hard it is to get a cat to do this okay that wasn't easy when you when you think about a container think about it as a file system snapshot well where does the file

system come from oh sure we're just going to do docker run and we're going to pull it from wherever from the internet how many of you were around some of you look like you were around when um docker hub was compromised and we were pulling compromised images from docker hub now we have to be careful there but who's going to decide where your file system snapshot comes from well we probably want to strip it down and if you've heard of cis benchmarks which i have the link for it coming up in a little bit we want to use the cis benchmarks for either docker containers container d we want to strip down the version of the file system that

we're going to run within our container that's number one container isolation okay we need to do it it's not going to happen by default so when we look at this the picture kind of shows us you know where it all comes from where we have our name spaces and our containers and so on anybody notice if i were going to threat model this where's probably the most dangerous point of attack on this slide anybody want to take a stab at it let's see if this works wait hey there it is my docker registry right there because if that registry where i'm pulling everything gets compromised and or it is not private and controlled within my environment no matter what i

do with kubernetes i'm going to be deploying insecure containers somebody dropped their watch okay so keep that in mind all right container best practices before we get into before we get into kubernetes the base image itself i already said it you need to secure the repo so you don't want to be using just some public repo somewhere sure when you're playing and learning and so on yeah but don't be doing it in production also cis is everybody aware of the cis benchmarks who isn't who has never heard of cis and there are some people in here go to cissecurity.org cis is the center for internet security it is an organization pretty much mostly volunteers and they work on contributing

to the benchmarks for best practices of security on the cis center you will find everything from security of basic operating systems of containers of kubernetes you want to start there you want to start securing your environment by using cis benchmarks in some cases it's difficult to do because you have to do it manually unless and i don't work for cis or anything you have to pay for a subscription and you get automated tools that help you do it in an automated fashion honestly i just suggest people support it and get a membership because it does make your life a lot easier okay please keep in mind don't run containers as root i still see people doing it when i'm

doing security evaluations of kubernetes environments or just of containerized applications i go wow look the container is running as root oh joy this is going to be fun coming visibility think about this you're gonna you're gonna deploy a kubernetes cluster it's a kind of a complex thing has a lot of moving parts which we'll see in just a second how are you gonna monitor all of that there better be ways to do it we'll talk more about that in a minute think of it as kind of a domino effect because what tends to happen when you're deploying out into a kubernetes cluster and you're using it to get extra stability in your environment so on or additional

scalability if one of the containers falls you can pretty much guarantee that most the containers are going to start to fall when you're talking about insecurity why because well you probably deployed those containers from the same place you probably used the same file system snapshot to make them secure or insecure okay privileges probably still one of the biggest problems when it comes to containers why because people end up putting out insecure applications that they get compromised then an attacker once they're inside they find a that it was launched via a username that they can then elevate privileges well there are very simple settings in containers that say basically do not allow any new elevated privileges and i'll show you

that in a second secrets management uh several years ago i i had some engineers come to me and they said well you told us we couldn't um keep our tokens out in github can we go ahead and put user names and passwords in the configuration yaml files that are out on github i said wait wait what no please don't do that secrets management use vault use something and please when you're doing a kubernetes cluster and you create some nodes that you're going to then deploy kubernetes to stop allowing ssh into those worker nodes because if you think about it i've got a whole bunch of nodes that i'm now going to deploy kubernetes on they're set up to allow ssh

you launch kubernetes and you have monitoring turned on such as kubernetes dashboard but if i ssh into the worker node directly your monitoring doesn't see me so i could then compromise your application and all this fancy monitoring that you have with kubernetes won't necessarily see me because i'm kind of running under the radar a little bit so once a system is deployed you do not want ssh on the worker nodes okay namespace limits i talked about it a minute ago we're going to look at some of these but we want to have them here is a great tool it's called docker slim docker slim will actually take a container and the application you run it through docker slim and let's

say a docker container might be oh i don't know 300 meg when it's deployed you run it through docker slim you're going to end up with maybe a 10 meg image that gets deployed saves you a lot of wear and tear in your environment so take a look at this tool okay often overlooked how many people know what se linux is how many people turn it off when they install a new linux server oh come on i know all of you do why because well you turn on se linux and you start to run applications and they start to fail they don't run anymore they can't open ports and things because sc linux has not been configured

properly i've asked other um you know sres and system administrators and so on why do you keep turning it off and they'll say because it's hard well um take some time learn a little bit about it learn a little bit about app armor one or the other but in kubernetes and in containers you can turn it on and it will help prevent things such as privilege escalation remember the number three problem with um config or incidents and remember the infrastructure this is ding ding ding for the win in other words your if you're deploying on an environment that is insecure by default no matter what you do to your kubernetes cluster it's still going to be insecure if i

have telnet running on on all my systems and then i deploy kubernetes on these systems well it's not going to be very secure so please keep that in mind make sure the operating systems that you're deploying your kubernetes cluster to have been configured securely okay so here's that picture again uh well not again here is the picture that describes the question that i asked a little bit ago where which is more secure a virtualized system or a kubernetes or container based system well we see in the first part you know we see oh we've got hardware we've got an operating system and then some applications running on it we now come to a virtualized system we

have a hypervisor level and then we build virtual machines on top of that well there is much more isolation blocking within a hyper or within a vm based system when you go to a container all you've got a container run time system if i can compromise this system then i can most likely get to all of the containers or if i can compromise a container i can very possibly get to other containers if i compromise a vm does that mean i'm going to get into the other vm no it's a completely separate operating system i could be running linux on one and windows on another people still run no never mind um so the point here is is yeah a vm system

based system is going to be a little bit more secure assuming that the hypervisor is secure that that is an issue yes i do take that into consideration uh but i really have a lot more ways to get into a container based system especially with kubernetes okay i don't know this this came out years ago but i still think it describes kubernetes in a great way because if we just inundate them with quarterly kubernetes releases hell it's it's just going to be you know security god they keep changing it um but that's what we're gonna do there are a lot of releases of kubernetes so let's see how it's gonna make things interesting so kubernetes in a nutshell we have a

master node think of that as the manager of all the things okay we have worker nodes and remember worker nodes are going to be the things that we want to make sure we we keep those very very secure we don't want them running extra services underneath the containers we only want them running the the parts of kubernetes that it needs we have pods pods are a very interesting concept and they are often overlooked when it comes to security think of a pod as an isolation chamber so i don't want to have a pod that is running both the web facing application and the database that is being controlled by this web facing application i want to have a pod

that is dedicated to maybe hmm oh maybe the web apps and then a pod that is is controlled for just the databases or maybe i'll have a pod that is specific to dev and a pod specific to qa and a pod specific to production why because they're isolation realms so i'll show you that in a second services services are all the pieces that we need to make kubernetes run and then there's going to be some very specific main components we're going to look at probably the one that we'll talk about the most at cd right there it is indeed something that is overlooked and often not well protected and that is the keys to the kingdom when it comes to

kubernetes so here's i like to say you know picture is worth a thousand words and it is in this case because it takes that previous slide makes it a little easier to digest because when we look at the master node and what we see on it that's where we're controlling it with kube cuddle uh or coug control as some people like to say but i'll say coop cuddle anyway we're going to control the master node and notice that xcd is actually drawn like a database and in a way it kind of is all your credentials all your things are are there we want to protect that we want to protect that like like it's fort knox you know put firewalling

around it and do everything the master node is where we're going to control everything that's where the scheduler runs we have the api that allows us to talk to it and talk to all the other things so we want that pretty well hardened our worker nodes contain kublets i i love all the names of in kubernetes it's just so fun to say kublets anyway but we have kublets that run and a common mistake in the beginning is that people will will do their darndest to open a couplet to the the internet they want to open it internet facing in some way it's hard to do it can be done but no that's what coup proxy is for

good proxy is the thing that's going to allow our traffic into the kubelets okay but see how the pods here this is where we want to isolate things because a pod is how we want to group our applications or our environments if a pod falls it should not take down the entire environment okay um whether it's dev or prod or who knows what it whatever it's going to be okay so you see you have multiple worker nodes um and then docker is running in there or container d it's all very similar so yeah that kind of explains or gives you a slightly better picture of what those concepts were on the previous slide but let's talk about

common sense and how we can now secure kubernetes okay first and foremost tls everywhere it is possible to configure things so you're not using tls huge mistake did i put too many exclamation points there i don't know i run into situations where i'm analyzing a cluster and i see connections and they have not enabled tls and my simple question is why and they say just didn't get around to it i'll get to it later something like that they never get to it later how many times have you said i'll get to it later and then you don't get to it i'll raise my own hand in that case harden the infrastructure remember i already said that okay

enable our back all of kubernetes uses role-based access controls you enable it with least privileges disable the asset based access controls this is you can do it but it adds a layer of complexity that just makes it much harder i'm not even going to get into it and you need to monitor everything i have an example of a great tool i happen to like it's open source called wazoo and they have a blog entry that they did on monitoring and auditing kubernetes that's one of the first and foremost things because most people that are installing kubernetes are just forgetting about the monitoring part because they think oh it's going to run just fine not a problem

and then this command here the cube cuddle get roll binding all name spaces every time i deploy a kubernetes cluster i'll run this in other words this is trust but verify i want to make sure that all the name spaces that i have configured remember the namespaces are part of your your containers i want to make sure that all my role binding and all my accounts and our back is is set up there are some tools on another slide coming up i'll show you to do that you could use a third-party auth server for the api in other words you could set up something outside of the kubernetes cluster don't use the built-in auth server for doing authentications for

your api calls hmm that gives you an extra level of security that you could put in i don't have enough time to go into all of that in a 45 minute session but think about that if you are in a really really this has to be as secure as possible i would definitely do that okay separate and firewall your xcd cluster the xcd make sure it is protected from any and all intruders and you have an extra level of monitoring on the xcd cluster if that is compromised pretty much keys to the kingdom rotate your encryption keys use something like hashicorp vault i like it it's open source it works great um linux security features i mentioned this

before but in the configuration of the pods there is an option pot for pod security policies where you can enable either se linux or app armor it's off by default enable one it's not that hard it will solve so many of your problems

kube cuddle get secrets this is one of the commands you want to run when you first install your environment well this is a better way to do it it's an easier way there are some open source tools that will allow you you run these it will kind of audit your configuration and it's going to tell you what yeah you've got some problems here um you've got some over privileged over privileged accounts or oh here's one without a u without a password or something you know so run these and make sure all your accounts are at the proper privileges they have the right access controls there are some now i'm not going to say it i i'm going

to skip over that part about github for right now i'll come back to it um your image repos remember the repos they have to be secure those are the um your docker registries okay keep those secure um all of the libraries i mean think about your supply chain where are the libraries coming from that that are getting installed here application i watched a team build an incredibly secure kubernetes cluster once and then they installed an application that they deployed which they had never done any security review of it they never ran any vulnerability checks on it nothing so the first time i i got into it i started looking and i went yeah but you

have sql injection right here on the application and i just got into the container and i can now do remote code execution and i've now taken over your entire kubernetes cluster and they were like how how'd you do that um because your application was insecure minor problem please don't forget the application just because you're deploying it with kubernetes and think about pods think about using them for isolation and i can i mean i can't say it enough also i mean how many is everybody here know about showdown if you anybody not know about showdown if you've never used showdan showdan think of it as the web crawler of all things internet it will go out

and i i use this in my honeypot talks as well because you can go to showdown and search for all the opened webcams of the world and and things like that but you can also search for um you know nsa uh toasters and and what have you but showdan you could actually go to showdan and search for xcd clusters that are open to the internet oops you might find quite a few because most of the time it's people that have just oh i'm playing with kubernetes i'll deploy something and they just take all the defaults so it's not very secure and yeah they leave it up there running and they don't realize it but sometimes you might even

find some production level kubernetes clusters that are completely exposed so i use shodan just for looking and searching for those types of things i like to if there's some way of figuring out who the owner is i'll i'll report it to them so don't be afraid to use showdown to look for your own stuff come on there we go configurations all right um remember most the breaches come from configuration issues here are your links and where you want to get the cis benchmarks for hardening your environment and i will tell you right now besides learning how to harden the environment you will learn all the configuration commands and settings and things like that by looking through the cis

benchmarks you will learn more about the configurations by just going through these than if you would reading some huge kubernetes book or something like that so take the time to look at first docker and then kubernetes look at how to secure the container first and then how to secure kubernetes okay so you get the idea i i don't know if anybody was here in in the very first talk this morning on the cissp but one of the things they talked about was what kind of a learner are you are you a visual learner you know are you a thinking learner what how do you learn things better or more easily and i i sometimes think

that actually going through these benchmarks i learn a lot more about whatever the thing is because it explains what you're really trying to harden it against what is what is the threat that this configuration change will protect against okay what about automation i don't want to put all these things in by hand well there are some ways to do it although my cat is frightened about automation what if you use jenkins or some tools similar to that there are some open source tools that will plug in to jenkins and they will take and do things like analyzing the image that you're actually deploying it will analyze the container configuration to see if the container is secure or insecure

so keep that in mind it's going to help you to deploy something that's very very secure or more secure than it would be if you happen to use jenkins or similar tools like that okay some additional tools let's say you you just installed your first kubernetes cluster you want to know is it secure um love aqua security they have cube bench this is something that's going to run and tell you have you applied all the cis benchmarks correctly or did you not apply any of them okay run it against a master node a master or a worker node okay there's another tool called coupe audit this will be a little more granular this is going to show you where you've

gone wrong maybe you forgot to to do some of the configuration changes that cis benchmarks suggested okay here's an example when you start to look at this you can see that if this works yeah there we go oops the read only root file system wasn't set see there's a common configuration setting where you can set the root file system to be read only we don't want them because what's on the root file system of a container it's going to have an etsy password file it is going to be out there we don't want that readable i'm sorry we don't want that writable i guess we do want it readable we just don't want it writable

but you see how it's it's telling you and you start to go through all the little configuration settings this is part of the problem of figuring out containers and kubernetes is the fact that there are so many configuration settings that's why i say start with cis start to go through the ones that you really need and then run the audit tools so you start to see oh mistyped one or forgot to set one or i set it to yes when i really meant no or something like that okay some other tools that are available claire this is a static analysis tool for the actual application containers very useful clar which actually integrates claire and your docker

registry so we can look at the whole package there falco i'm going to show you falco in a second falco is perhaps my favorite thing falco looks for weird stuff going on in your containers it looks for abnormalities um we'll see that and then coop hunter is how you can actually do kind of a pen test once you're done and you've configured your environment you can run coupe hunter against it here is oh wait did i oh i'm missing a slide here i had an example of falco sorry i had an example of falco running um but i had to take it out because i was having weird problems with getting it to to sync up with the video

but yeah go to the falco.org site you'll see it's an agent that runs within each of the containers and basically it looks for anomalies it looks for suspicious things like if it sees somebody is doing cat or or something is doing cat etsy password um yeah that's probably not something that should be happening inside your container and yet i have had vulnerable kubernetes clusters and the containers where falco actually caught someone doing cat etsy password it was a red team and we caught them we were the blue team at the time and we were we kicked them out and they were like ah that was how we were going to get into your systems and make you look bad

but yeah anyway all right if i were a a hacker i am hacker i i'm going to attack well how do i actually approach pen testing or attacking a kubernetes cluster first thing i'm going to start with is all the services that run on all the ports so there they are i would start by just looking and making sure first of all does anybody notice anything that's missing here that that might be important what might i look for in an attack methodology within a kubernetes cluster there's two things that are missing

what about before port 443 what about port 80 what if i find an open port 80. oh somebody forgot to turn on tls what if i find port 22. oh crap let's turn that off okay keep that in mind so this gives you a kind of an attack vector obviously um 6666 for xcd i'd be looking to see if that is exposed in some way okay so it's the same kind of thing you know if you're if you're an attacker and you want to go after a kubernetes cluster these are the things that i look for can i get to the api server if i can get to the api server i have an attack vector that oops

probably shouldn't be there especially from very depending on what network check for fcd make sure that cd is locked down if that is not then you have a problem checking on the kublets the kublet port should always be read only it should never you should never find a kubelet that has right access to it okay the container vulnerabilities i mean there's nothing stopping you from running a quick vulnerability scanner against any of the applications that are running within your containers do you have sensitive files that are not protected remember falco you actually will configure file code that say oh i have these specific files not part of you know etsy password or something maybe i have some other files that have

some important things in them that have to be part of a container application so i want to tell falco hey keep an eye on on this file or these sets of files or whatever kernel exploits there are going to be vulnerabilities in the os's that you're deploying via kubernetes so if one comes out please remember to patch the systems and then redeploy your worker nodes which is usually very easy you can roll that through but i do see people forget that a lot they they don't remember you have to patch those containers you have to patch the entire cluster because if we can get privilege escalation we can take over your cluster the vulnerable apps yeah that goes

without saying so some final threat modeling to think about when it comes to kubernetes think of the entire ci cicd pipeline if you have one if you're in the cloud you might have a very automated one might not be an issue think about your workload security think about pods i usually say this in a honeypot talk that i give that 90 of the time you do with honey pots should be planning same thing with kubernetes kind of 90 of your time should be spent thinking about how can we group applications such that if this cluster or this pod is compromised what's going to happen what's my risk at that level container security directly file storage security keep in mind that

in the kubernetes world file storage is not permanent unless you set it that way it is transitory so if a container goes down or the cluster goes down you will lose whatever storage you might have had in there unless you make it secure by default or permanent by default the network security remember the infrastructure and then the application security itself those are the things that we really want to think about when we're looking at the entire package okay so your key takeaways for today okay common sense is still there um i like to say in all my talks that we keep spending billions on fancy software and hardware and all this stuff and yet breaches keep going up

why is that because we're forgetting best practices and everything that we've learned doing security so stop overthinking this stuff go back to best practices secure the infrastructure have a secure repo you know don't let people get in and play around with your repos and things like that okay stick with the basics okay stop the fancy for god's sake um just use cis benchmarks um that is going to be very important and then our back and all of the configurations we want to make sure that we're applying benchmarks across the board okay if you take one thing away from this talk today if you learned about cis there's a good one threat model your environment okay

the access controls i mentioned this more than once because it is often what is overlooked in deploying kubernetes clusters okay did i mention our back yes because it like i said it's overlooked patch patch your environment log for jay just hit back in december how many people had to patch their kubernetes clusters because they were running java based applications it was a little easier because okay you're in a kubernetes cluster you can roll through the containers and get get all your pods upgraded and so on but you still had to do it logging and auditing if you don't monitor how are you going to ever know and i throw the wazoo back up there again take a look at it it's

just it's a great open source tool and it works very very well with kubernetes and containers okay so that's it let's see did i make it oh barely thank you very much i will make my slides available if you watch my twitter account um i'll post when i post the slides to my github my github is also rainbowcat same thing that's it any questions [Applause]