Hacking Kubernetes 101

uh it should be visible now okay perfect thanks okay so well first of all uh thanks to besides to christian to you david to all the organizers of this conference it's my first time here and i'm really excited to present well the weird things i like to investigate about so this is the title of the of the talk it's hacking coordinates 101 as the leader was saying this will include like some key ideas of how these can be attacked and also how uh it could be hardened to avoid those attacks uh and i hope well you will all like the talk so let's start well who am i my name is daniel monton aka star code

i'm a pentester at intel innotech security i like to hand cvs from time to time in random software i download i've written some tools like that one and the expounder and you can find me there on twitter if later you have any questions or whatever so let's start er first of all as this is a 101 we will start by introducing what is kubernetes for the ones who don't uh well don't know the technology that well maybe you have heard it but uh you know this is a more in-depth introduction to this so this is the oral schema these are the pieces of a kubernetes environment all the nodes maybe virtual machines maybe physical machines

and mostly their worker nodes which are like this ones and the master node the master node is the key to the kingdom if you can compromise the master node that's it you get everything from there and these are what the parts of of this the master node includes the scheduler the api server which is actually the more or less the heart of a kubernetes environment as you can see everything interacts with the api server it's like you know the if you get into this you're done it has an optional cloud controller manager in case uh well you want to in case you want to launch or start building a kubernetes environment uh well in whatever cloud services you want

uh and this is the well a typical worker node uh this cubelet is an agent uh which runs in every of the nodes and interacts as you can see also with the with the api server and so uh well dc advisor thing it's uh like a matrix uh a matrix thing that sometimes uh leaks information is full to an attacker and this cube proxy is the component responsible of exposing services to the internet and which is well as you already know er probably the weakest point of this and in each of these nodes uh there are pots it may not be equal a part to a container but in the majority well most of the times

there is just one container per pot which is like more or less another container and i forgot to mention the epcd database this is a key value database [Music] it's well it holds a lot of juicy information uh and we will see it before we'll see it after sorry uh and the runtime which is uh what what actually makes the container park uh well it makes it to work so after it was before it was a docker uh but now they are changing that to container t and because docker is actually built on container d so yeah it was a nonsense um okay one second and well and as you can see it in a

different structure and these are services which is the first and almost the most not vulnerable but one of the most critical components of kubernetes environments which are well the services uh that may or may be not exposed to the internet depending on the type of service and this is well uh where every attacker almost almost every attacker should start looking into so this is another another slide in which we can see more or less the different attack vectors from zero to start getting into nodes start getting into boats these are these are mostly the well the most used techniques uh actually probably is this one and this one this one are well and also yeah and the fourth one

and the other ones are less common but let's start explaining so the api the api server which i told you this is the the heart of the kubernetes environment and this can be interacted with uh well it is an http api so you can interact with it with cool or wgets or even with the well the command line interface which is cuba ctl the xposed dashboard we will see it before after sorry uh we will see it after and it's uh a very well a very stupid way of getting phone so the cubelet api as i told you there's a cubelet running on every node and this also has an api and well it's really easy to

get the code execution into any of the pots of the node using the cubelet api and the last of all of these techniques uh the etc api which is this one this one is way harder to exploit but sometimes it is exposed so from an attacker point of view this is what you will see if you are in a well if you are interested in something and you suddenly discover these kind of ports this is what it will make you think you are in europe testing kubernetes environment so these are the the common ports here you have the uh the components uh all these ports are from etcd the the cubelet apis what all this stuff okay so the cluster

components and this is not necessarily like this because these are the default ports in which kubernetes listens or a node can listen [Applause] like a service so you can expose something to the to the internet this is not necessarily like this okay but the the rest of them yes um okay so container metrics this mini cube thing it is rare and not really common and because mini cube is a like a self-contained kubernetes environment in uh in virtualbox okay the the cubelet apis the cube proxy which is the one interacting with the well exposing things to the internet and this is just one more custom things okay so first of all is kubernetes hacking even a thing uh well

yes uh this is an attack matrix uh similar to you know the mitre the very famous one meter did this time it was microsoft who did it and it was the past year even even wasp has started to to write i know if a top 10 or something similar as you can see many of the techniques we mentioned before and that we will see in detail right now are mentioned here uh for example uh let me see what uh for example well the exec into a container the dashboard for instance uh or well any service uh an exposed dashboard okay so application vulnerability leading to compromise etc and the cloud thing all that stuff so

you know in case you need a reference if you are in a pen test or whatever that you need to know about kubernetes to to do your job then you can always check this to get some some fresh ideas about what to do next okay so this is the kubernetes dashboard whenever you create a kubernetes cluster you get one of these one of these websites the and the bad part of this is uh well that many people actually don't protect this in any way and you can go to show them and just well check if there are exposed dashboards go you know here to workloads but click on the pod and exec and you will be granted with a

with a nice cell in whatever pot you you want so this is terrible and some companies have been well compromised using this attack vector so we need to take care of that in many in many cases so uh this is the other this is the other api well one of the others which is the cubelet uh as i told you there is one api uh per node okay so maybe you cannot interact with an api in one of the nodes but maybe you can interact with another in other node of the of the kubernetes environment so as any of the apis you can interact with it using cool or level you get or whatever typical linux command you are used to

and also using a command line interface this one i think it's non-official i think so but it works decently so er what can be done with this well directly you can well get a list of pots running in a node okay with this command and after you can uh well you can run a command inside a node you need to specify uh the namespace here um but it's fairly easy to get command execution into any node sorry into any port of the node so this is a you know a very easy vector and if you expose this port to the internet well whatever apt that's using mascara can can detect this and well you may have a problem

so and also you can check the logs which is interesting and you can check the logs and you for example can search for the cube edm well this is the cube idm is the command line interface used to well when you create the cluster you use this instead of cube cdm okay so you can find the token and use to join the machine to the to the cluster and you can just add your own malicious machine to the cluster which is not good at all and also if you are using http [Applause] to visit whatever and you are using cookies or tokens or whatever uh or ids and you are well putting that into a get parameter

you are well all of this can be exposed just by making some requests to the logs to the last thing of the cubelet api so here's an an example you can see it's not difficult to do this okay so the last one the one well this one it's way more difficult to exploit because you need all these files the ca crt the health check one and the lc client you need to specify the api version with this environment variable there is also a command line interface by the way and well you can just for example launch this and start uh well start dumping the whole database of the kubernetes environment uh you know the good thing about this is that you

need these files so uh as i said this is not that easy to exploit but it's another way so we are in in a po right we have compromised uh whatever our web application exposed to the internet whatever a typical scenario we are there so what do we know uh okay so now uh we need to enumerate a little bit both as you would uh normally enumerate a machine after you compromise this and also taking into account the particularities of kubernetes environment for example you can check the environment variables and if you see the kubernetes in the world kubernetes you know you're in a kubernetes cluster also you can check c groups and if you see

cubots as well you are in a kubernetes cluster and this is interesting and this is a very common uh misconfiguration that administrator administrators do sometimes which is not disabling the service account mounting on a pot a newly created pot so what is this when you create a pot many times if you don't disable it this is enabled by default you can just yeah well if you get a cell you can just go here and grab the access token for the default service account and with this you can interact with the api with the api the well the api server which is by far the most important api so uh like this is free themselves almost

um also you need to check if it's present again the binary of cube ctl or wk or cool to do it manually credentials ssh keys database whatever many times they have databases local and well when there are databases in you know in localhost maybe the the web application you are you compromised or whatever it's reusing credentials uh for the rest of stuff uh like uh i don't know the root account in in case you are not root directly on the container uh whatever also use tools like you hunter try to accept to the rest of pods or modify all of these types of objects and and also typical stuff like trying to see gain more knowledge of where are you exactly

what services you can access since traffic etc also you can if you have cube ctl you can try to well with these commands you can try to um get like a more exact idea of what permissions you do have and well if you have a lot of permissions you can do this which is what changing our role binding which i i will explain uh later in the end so this is an example of well an attacker gaining root on on a container and let's see well what we asked so far [Music]

okay so you can see all the environment variables so we are going to coordinate this environment

okay [Music] so there you see the um what i told you before the service account that's well automatically mounted on every port if you don't disable it of course so let's see

yeah so that's the access token and with that you're able to to access the ap the api of the api server okay so there's no cube ctl which is normal in a in a container okay the good thing in case you don't have er well cube ctl and you don't want to learn the whole api and you don't want it to do it be a cool you can just upload it like i did in the example uh i know this is a linux machine with 64-bit architecture so i just download the the one i need i get permissions and from there i can start checking uh what what can i access

okay

yeah so yes i can leave spots now we'll see some more examples but there's something important to take into account yeah that is that in this case it wasn't even needed to to use the token where we grabbed before i mean we can do all of this without authentication which is terrible because almost anyone getting god execution in any of the containers they can do whatever whatever he wants so

okay so can i list notes yes

and i get notes details of the notes okay so as you can see uh we asked the api uh if we can perform various actions and we can so well this this would be a terrible scenario for any admin okay so let's go to the secrets part okay so secrets in kubernetes well they are just as you can see here this is a screenshot from the official documentation they are just basically for encoded strings so this is almost equal to clear text and you know anyone with api access in this case as you saw anyone could access the api it can retrieve the secrets so uh we could do it if we didn't buy any reason have qctl

we could do it using this okay uh using cool just no using the api this is what gets done [Applause] by cube ctl in in the background so in this case for example uh we wouldn't need this header but in case for example that we couldn't uh all all of these uh commands return no we could just have added the access token we got from the well insecure mount of the service account token and use it here and we can get secrets and we can well exactly check what's in the in the secret object so here is an example

[Music] okay

okay

perfect so um here yeah that's the value of the natural value of the secret

perfect so what you see here is the well but it's a story in the secret that you can see that well secrets in kubernetes are not that secret so it's something to take in talent okay as you can see here these are well this is another command which is listing the the namespaces namespaces are well logical divisions of clusters a physical cluster like can be divided into different name spaces uh the default one is this one but there are others created when you well when you create a cluster like this one the cube system one and the cube system one is particularly dangerous because all the secrets belonging to this name space yeah well [Music]

are of service well they belong to service accounts which they hold very high privilege so in case you have access to this namespace and to the secrets and well your remote zone you got everything so this is what we did before without specifying the namespace as you can see this is the same result we got in the previous slide and here we can list and check the output of you know we can list the secrets of another namespace and here we can even get the um the clear text of uh of the secrets in in other namespaces so this one is critical yes the no that's the that's the point of this okay so uh a part of xposed apis how do we get

on uh well with this with services uh here well i don't know why this wasn't working okay but here you would see in a normal situation you would see the external ap or more concretely here you would see the external ep of of these services okay so there are many types of services okay but the ones well the most common ones are the cluster ips which are responsible for well just launching an a service a network service in inside the cluster so you can access it like well something like if you were in your uh at home and you open a netcat okay that would be an open port but yeah but if someone tried to reach the board tries to reach

that board from the outside uh it wouldn't be possible but the note ports and others okay but the note ports are the most the most common ones uh well they are used to expose uh services okay to the internet services uh that could be whatever from ssh to um i don't know to a web application and whatever you can imagine so in this uh well in this command it should be possible and i i don't know exactly why this time when i did the screenshot didn't work but you should be able to see here the external ip so if you don't take care of this and well if you don't do a proper not i don't know how to say this

if you don't uh have a clear list of what you expose to the internet uh it's and also when you have a like very big development environments and so using kubernetes it's very easy to well to expose something you don't want to expose so more attacks okay so we're in a pot but apart well where the where in the container of the po okay so but we are well we are like if we were in a docker typical docker container we have no no more permissions okay but and there's a thing which is a very typical attack that if you are well first if you can find a bugs in the world you are uh well

in which you are you can try to well access another boats etc and this uh this permission that we listed before and we have it um well we can create bots okay the way of declaring about is with these files if you declare this okay the host volume and the amount path and also the privilege true what this would do is to create a bot okay and mount the whole notes file system into this folder so we could just exec into the into the attacker port in in this case and if we go to slash attacker we could see whatever what whatever it's in the in the host's file system and this attack is particularly

dangerous because there's well you can specify exactly in which node to create this spot and there's uh well a high chance that an attacker will try to create a malicious spot in the masternode because one thing you are once you are in the masternode uh well you get access to everything because uh for example you know the token i mentioned before to join machines to the kubernetes cluster it's there they logs everything they access tokens everything's there so and the dtcd everything so once you get this you're done as well okay so what more to look at once you are checking the security of the bots well you can check all of these options okay

well the privileged true one which is the one that actually allowed us to do the previous attack the host ipc the host network this allows the pod to access the same network of the node there are three or four options and in different combinations can lead to multiple ways to compromise the the kubernetes environment so and here is a very good example of of this okay well they put security admission what uh this is a way the bot security policy this is another way to to declare the security context of the pod now well now that's deprecated and it's used in if i don't remember if you remember correctly it was an admission i think an ambition

controller that i was using right now to well instead of the bot security policy okay i also for those who think they are safe in the cloud here you have well some examples of how to of how to leverage well a simple post compromise to what to get all your only assets you have in your aws organization or your google cloud platform or if one for example well the this is the advisor thing i mentioned before this is not particularly useful but for example this one the aws one is very typical because right now there's a what a very popular service aws is providing that it's aks eks or eks whoa um with this one just to run a

kubernetes cluster in the cloud and from that you can both do like a double that will compromise both the cloud environment and also the cluster and the package manager this is the typical package manager using kuerner's environment it can be attacked as well and here you have an example this is we cannot stop as much as i would want in all of this but in case you later won the the slides just tell me so you can get all all the reference okay so for instance what we could do if we have a kubernetes environment that's running on the cloud okay so for example uh this uh both the master node and the worker nodes in aws kubernetes environments

are ec2 machines okay typical ec2 machines so uh if you well maybe if you if you have ever just like tried that bounty program or read some back bounty write-ups or whatever you may recognize the dc2 instance metadata service you know the 1 169 dot 254 you know that ap yeah so this uh service well you can get credentials from this and this is very typical and from this you can well try to escalate privileges and get well into the root account of the aws organizations and same thing with the the cloud metadata service software or workload platform the same thing okay so uh about the permissions and how they work well and you're gonna specify uh

permissions based on the node you want to [Applause] to run the bot in okay this attribute one it's not that used uh or at least i don't know many people who use the this type of authorization the role role-based access control yes and we will see it in the the next slides which are more focused on hardwaring okay well and the web hooks as well here you have the documentation if you want to to read it more in depth the web hook and a well are used but you know mainly people use this one or at least that's what i see from the people i know they use kubernetes okay so hardening tips [Music] for those who are developers or

system administrators whatever okay so first of all uh depo the prosecutor mission and norwich polar images there are various admission controllers which are emission controllers are pieces of software between the api server and you okay and there are some of them which can actually help to to secure well the kubernetes assets you have network policies these are useful as well but if you go establish this kind of policies but you don't for example but you specify this well you may get into trouble because well if you have some policies but you allow the attacker to access the network of the of the node well uh no not very good idea uh disabling the amount of the service account token what

we saw before uh well it's a simple command okay to get this done and greet the secrets uh well if you don't want your stuff to be in base64 formats then uh do this disabling the anonymous access to the api server just this would uh actually make very way harder the all this stuff that we did okay okay also disabling the unauthenticated access to the cubelet apis this is well important okay because if you have some critical information in some of the pots uh and you don't disable this well then you're done and quite a lot of recommendations so what i mentioned before about role based access control is based on this okay this is our these are um

well the typical objects used for role-based access control which are roles which are like the permissions part okay so where what you would have access to and role bindings are just the object that will bind a role to a user and this is the same thing but a cluster level but at a cluster level uh okay and uh well for the typical recommendations don't assign the cluster admin or the admin role to any service account okay these two are critical don't just go on this is like giving root permissions to uh apache server it has nonsense it's a nonsense um okay well don't grant the the permission to escalate to any service account so in also when you're declaring a pot

you shouldn't specify the privilege as true nor the host ipc host network all the options we saw before and if you don't need to well you don't need to write anything you can define the whole container as a well a read-only file system okay and you want to specify also the permissions everything [Music] also well [Music] this is important well in case you don't want to to expose your api uh i know some people that may need to expose the the api of the api server but if you can avoid it you can do it modifying this okay okay well so this is uh for the cubelet api same thing sorry okay okay okay yeah i'm finishing i'm i'm

almost there okay so uh also well protecting with a logging and the kubernetes dashboard [Music] also the containers you want to use well this well this has no sense because now kubernetes doesn't use a docker but check the well try to harden also the the images uh your juice you're using review the chrome jobs this is a very the jeff's underground jobs are a very common way of persistence inside the kubernetes cluster unchecked this is uh these are all the type of services which expose well expose a network service to the internet so review these ones uh okay and this one not that very important okay and for pentesters this is our these are the tools of the

trade okay there are many many tools uh i like this tool called striker and qv scan but here you have a lot more in case you wanna play around with all of this and that's it this is these are all well all the objects which are in the in the api uh so if you want to take a look on the security of some of this we have seen the services service accounts secrets whatever pods but there are you know security risks in others and this is a interesting thing to to take a look if you want to if you're curious about the topic so that's it i hope you like the talk and whatever

questions or doubts or whatever you want me to do well here or in twitter if you if you prefer great thank you very much uh daniel i don't know if we have any questions for in the meanwhile just to give people a couple of minutes to formulate those questions um i have perhaps like one out of you know the curiosity i was really uh curious about i'm really cool i was really impacted by the one that you said about the vulnerability that allows you to mount um you know the file system the hostile system into a pod by using a privileged container right yeah is this is this kind of like a default behavior i mean would that by

default yes yeah right yeah right because i remember that you because i remember basically by default you can create privilege containers right and then you have to explicitly disable it right but if it's not if it's not a privileged container would that attack still working uh be a thing uh i think it wouldn't work i think it wouldn't but uh anyway uh as i i think there are no politics uh that like would like uh make an attacker not possible to define uh apple that's a privilege i mean if you have the permission to create pots i don't think there's going to be any permission to create bots but not privileged so once you get the create pods permission

then you're done [Music] okay great thank you very much uh daniel i don't see any questions and again you can reach out to him on twitter but also on slack channels we'll be there