Securing & Attacking Kubernetes

thank you all right hello everyone I'm Vincent as introduced I will talk about the securing Cuba needs but a little more about how to attack human needs because I think that's it's more fun too so first a little bit about me I'm Vincent I work at kpn it's the Royal Dutch telco in the beginning I used to work as a Redeemer and it's decided to switch to the blue team because I think is good sometimes to look at both ways you know how to defend how to attack normally I love to do body stuff but the Cuba needs is a lot of llamo and I thought the other one was interesting too so first I will talk a bit about

what Cuba knees is then I'll show you the tools you can use or the tool actually how to interact with it and then I'll talk about some common pitfalls and show you some demos on how to actually own a closer or own accord and then I will talk about how to secure your cluster I have to say that some of the things I talk about in this talk have already been 16 new releases of Cuba needs because Cuba needs really move fast I think when I started this research it was version 1 8 and when I gave my first presentation is already version 112 so they are going really fast and they're starting to think about security now

they have a lot of secure by default and it's actually pretty good I do have to say that all you can still find nice yomel configuration files to put your Cuba nice cluster back to the old easy to manage way so there's still be a lot of available clusters out there so what is Cuba nice but as I said Cuba needs is mainly llamo it's written in gold and it's a container orchestration and management tool created by Google I think they release it in 2016 I have 2015 of 2016 year no 2014 sorry guys it's hard to read sometimes and yet mainly use this blocker so all of the issue should have the docker like there

containerization previous escalation things they they also work in Cuba needs and and vice versa Cuba needs you can now have on AWS ec2 you can have it on the Google cloud it's even the canonical oompa guys have their own one and it's with with cloud support it's really easy for you to automatically scale your your applications you can automatically assign external IP addresses you can automatically create load balancers you can create replicas sets of your website so if you have a lot of people coming to your site at one point Cuba need skills really nicely and I think that's the reason why a lot of people are such a fan of Cuban AIDS I don't think it has anything to do with

local Cuba needs also means helmsman or pilot in Greek so the best tool you can use for talking to your cluster is cube CTL keep the couplet is actually a REST API so you could also use curl and JWT tokens if you really wanna you know eight yourself but this is the best tool and it's also a really good post exploitation tool and you can download it statically from the key Winnie's website so you know if you ever a cube knees closer you just get this binary and you're good to go and it works a lot better than using a curl so Cuba knees has namespaces with the QCT I'll get namespaces commands you can have a look

at all the namespaces that are there you can use this to determine the scope of a user or a pod you can also use it for administrative boundaries or resource restrictions so let's say you have a Cecil compartment the only one Cecil to use one CPU because you don't like C so people then yeah you can do all that you can make policies that like I can only talk to people to see so I cannot talk to people a lot of departments so you can actually use it to find great your the role based access control which is what they actually have in Cuba needs it can get quite complex with all the llamo code that's involved your cue needs

words with pods I thought this can be one or more container it is usually this were yeah so a port can be a website hosting or a micro service and it's also just like everything is configured in your MOU over here you see an example of using the cheap CTL command to list all the ports in the cube system namespace and then you can see like there's a dashboard as a scheduler there's an HDD which contains all of your secrets and you can download a lot of images readily available and it's quite nice if you have a nice repository you can create your own repository put some images in there just make sure you maintainability

availability management program for your images as well this is a nice example of how to create a pod it shows that a value version us use the kind of the titles the thing you want to create then the name of the port the namespace you want to put it in and then you can say like ok I want it to be a container running a wound to 1404 and because a pod dies when it's done with its job you can make one that runs in an infinite loop so it yeah it doesn't die I can and then genex web image for instance will never die but if you have a port that sends out an email the moment it send

out the email or the port is done and then you have to clean it up or you can restart it if some some things are triggered you also have cube with QC TL exact which i think is a really interesting it allows you to communicate with your with your pod and if for instance you want to open a shell on a port in a namespace countries that's called Nepal you say like great QCT exact - I for interact a TTY for spawning a TTY so you can use sudo and pass a video know whatever and yeah there you go yeah you have a show this is the more important thing if you're you want to attack a closet because it's

pretty useful to interact with it alright cubies also have services so let's say you want to host a website for instance and you want to expose it to an external IP again if you use a cloud a cloud Cuba needs then you can automatically assign it an external IP and you can easily find the services with the get SVC command because CDIP that's associated with it and the type of fording it does this can be really effective for finding attack surface so let's say you have a port with not many privileges you can listen the services that are available you can have a look some of them might be extremely ritual some of them might run in privilege mode

in which I'll show you later I can cause you to completely own cluster and it could be really useful and yeah keep CTL works with secrets secrets of sort and SCD I still think they they do it in plain text it's in a binary format but if you want strings you can see a lot of the token snow and a lot of tokens are created automatically for API access so you can see there are some service accounts in there this is the fault token I created my own token then you can also add SSH keys yeah all sorts of Secrets actually and you can use these secrets again to find in your your role based access control and as I'll show

you in the advice later it might be good practice not to automatically mount the service account on each pod because having a service account means that you have certain privileges inside the cluster and if you manage to own a clock a small port and you can then talk to other ports then yeah I think that's an issue but yeah if you want to have a look at the secret tree you type that command so now what what could go wrong what have I seen I'll talk about that a bit later but I usually have yeah what I have seen is the exposed end points under strict api's there was a certain period where Cuba needs had no authentication or

authorization everyone who could talk to the Cupit API was opened by default which was funny this is actually what I found in 2016 as well sometimes you have previous containers running with a command injection on the outside and the moment you're in the pod yeah you can own the entire cluster and also the dashboard has been unsecured for quite a while so for me what's what intrigued me when I started working at the red team was I I found a cube in each cluster in 2016 when I was scanning the KP an external ranges and k10 is quite a big company and I was actually happy to find something so I found this thing I'm

running on port 8080 and I had no clue what it was so I do they're busted things I found some end points and then I googled them and then I found out it was Cuba knees and it was indeed a Cuba needs yeah two years ago it was no vacation I was up in already I was like really happy with it so I created the pod and with that thought that was in the internal network I did some scanning and you find the Jenkins and you know then you know it's over and then I found out it was a customer and then I just stopped doing everything but yeah they luckily when you look for Cuba needs on show down you

can find by 16k Cuban eat masternodes there a lot of them are protected by basic identification you can fight with one set but yeah nowadays they're actually having authentication which you know think it's really good there probably some in there that are not authenticated but if you wanna have a look yeah you know showed up and later I'll show you a more interesting one I actually managed to found an unattended Gator Chinese dashboard you know it's like yeah let's yeah let's not attack our allies let's have a look at our you know our enemies because then I can say in my own country which is nice and yeah this one was not authenticated and yeah

you can you can look around if you want you know it's the Chinese also interesting is at CD what I just thought you liked s CD is the holding its maintains your secrets and you can find a lot in China again they're like the number one and it's like yeah you can try to run for instance that query there and because that CD by default has no authorization I think Cuba needs now does but if you you know make a boat to just create all the keys you can find AWS keys you can you can yeah I get jaded with you tokens may be hijacked Acosta with everything you find in there yeah it's quite interesting so yeah now

let's talk about how to know what to do when you are inside the pod this was my example when you know after I found the cluster in 2016 I did a pen test and they were like oh yeah you know Cuba needs do you want to do a Cuban in specialist suggests sure so they gave me a guinea an API endpoint with a command injection misery so I could just type hostname and I get the output over there I created a quick quick shell in Python and I got an actual reverse shell and when I got my reverse shell I typed print n you can see the cubed API this is the the one managing your pod and if

you do that you can also try to listen secrets and as I said before the deserves account is automatically mounted so in here I can find out my name space I had the client certificate to talk to the to the QBs cluster and I have a jadibooti token which means I have some kind of privileges on the cluster yeah and then I net color to the cupid it looks like okay sure I can I can talk to this guy so now I have a small demo showing what you can do when you have a shell you guys know you can zoom out it is pretty impressive there you go so I start with a shell but this is actually all my own

person oh wait that's not readable is it [Music]

[Music] swaggin one more it should be fine so yeah I start with a shell over here you see like the name space I mean this I listed all the name space and all the posts are available my left you see the name space will right you see the pot you see there's the QBs dashboard available there no outside get services for all namespaces and you can see there is like available service available there are some human needs clusters that are available I communicate with and what I now try as I tried to spawn a container if leverage mode that runs nginx because nginx doesn't complete so I have all the time in the world to try

and do a pivot so because there's a lot indicated on the authorized cluster I just created this new pot as you can see it's up and running and now I actually have a root shell inside this pot which is running in privilege mode so now I go to slash there and you can see like all the character of what devices that are usually only for the holes that are available and this amount with star1 which is the both the fires of the cluster so I just mounted you go to start - like hey looks like a root filesystem let's have a look at positive Edie yeah you basically there you can also cut pickup eat see shadow you can you

can add a cron job you can you can put your authorized keys in there yeah this one is actually a Donald Trump later I'll show you another one where we actually look shadow file so yeah what just happened yeah I would be worried because I think the p-type is on that particular cluster yeah what happened so first I I use exactly spawn a shell and when I had my shell I did some you know some recon I had a look at the faults in the service they were available and because I had so much out food it means that yeah I haven't at least me taxes on on everything in the club means closer for in his name space but since I used

to all namespaces yeah I could see everything in the cluster so then I created a privileged spot with privilege mode enables and mounted the root filesystem and from there on it's only the filesystem so you don't see Prague you don't see sis but if you do a cron job or you add an authorized key and change the SSH configuration at yourself in the shadow file you basically completely alter the couplet note so the problem yeah the problem is actually privet mode you can see I've highlighted it right there there are some valid reasons for you to have this privilege mode enabled if you want to do some stuff on the filesystem then you might need this but if you don't need it

please hurry up because it's a security risk over there cuz privilege mode allows you to basically do the same things on the note that you can do on your own file system so if you have root on the Kuban in support you actually have root on the node as well and if you then can mount the filesystem then your root in the filesystem so it's the same with poker I think in with other containerization tools just be very careful with the privileges you want to give it so during the pen test they fix some issues and but they still had a Cuban nice dashboard available and before version 1.7 of Cuba needs the dashboard was often by default there was

no authentication so if you manage to do some forwarding magic and get to the dashboard you can just yeah basically do the same thing because it was too close for admin nowadays you can use the cube config file or jadibooti talking to authenticate too but in my demo because this still happens a lot that song can even on the Internet you can still yeah pretty much own on the cluster so what I will do in the next demo I'll I'll try to list all the pots this this time I'll do it in one one one in space and I'm not allowed to create anything in this namespace because I cried to create the same pod

try to get a shell and yeah it doesn't work because I have no privileges but you know I can still this is services in the keep system in space I found the first queue nice dashboard so maybe let's have a look at the Cubans dashboard you can do some curl and then you find like oh no authentication this is a Cuban each dashboard and then you can start to you know interact with it if you don't have read access on their own the services for instance then you might need to do it use masks and or and mop to find anything else else in there but the ranges can be quite far from the power part so you might be a bit unlucky

there so now I'll try to give a demo there as well so yeah we you try to get a live show in Nepal this time with listening spaces and you see like okay this is the wrong demo is the right level that's thank you yeah yeah you shouldn't work this you will try to do a privileged one yeah I'm forbidden I cannot create any pods here so what I do now I listen these places I see that there's a cluster IP or the Cubans dashboard which means if I connect to that IP and talking to the Cuban is network and now she's in a demo video popping up ah yeah there you go so yeah

now I'm on the dashboard I did some IP forwarding got access to the dashboard and I can look everywhere I can you know go to different namespaces I can look at tokens I can look at the secret token I can look at the certificates I can look at the next page the nice data with each other you can basically just browse around there are quite some ways for you to get access to a port over here I will just do another example of spotting upon but you can also try and find it already this container you can look at the adjacent configuration of each individual pot and if you find that one of them as a privileged true you can

just interact with that pod and if it has a shell installed you can just do the same trick I just showed you I'm looking at the cubing is that sports togas yeah it's also really funny that the Cubans actually has a web-based shell for you in the dashboard so you can you don't even have to use a reverse shell think I'll show you now like okay this is a log file and yeah I'm gonna copy and paste the same llamó file I just had but the only difference is that I already mounted the root filesystem on slash food because my slash food you might put is empty so I decided to mount the hosts file system over there and yeah I do a

privilege mode mounted yeah that also works he's pretty funny and yeah in this life you directly I call it boots because there's a lot of loot over there so I'm spawning the ports now click on it and then there's an exact button over there in the upper right corner then it shows your show it's just three eyes nothing working all browsers but it is or does you can see spirits mode so yeah there you go and look at pop city look at the shadow file my passwords in there and it's a whole folder and a home floor is a double trauma therapy there might be some interesting things there is a lot of evidence there

I hope the FBI isn't watching this video so yeah that's interesting right the Cubans that would now has authentication it's not open by default but for a while on the Cuba - were on the Cuba news website they were like hey you want to go back to the old configuration here download the Java file they find removes it from the website because it's a bad idea to not have authentication on the Cubans the dashboard my another nice example the four thousand ten thousand two hundred fifty was actually the port directly talking to the cupid api so if you had that port exposed on the internet there was no authentication up to I think February of this year so if

you find this port open on the internet and trust me you'll see my later slides there are ports open on it you can get unauthentic code execution as the admin on all of the ports that are available so yeah there's a nice medium post here of a guy who found out that I think of him on Nero - on the squad sir and then he went through the logs and then I just saw some guy doing a comment like this basically doing a touch mob mad I think it didn't help you get with his money real minor and yeah you can do it in all your ports I mean if you have a big Cuban ethos you have a lot of mining

power so if you have a VPN I think you should have a look make some money row there's also a ten thousand two hundred fifty five board and Cuban eat which is the read-only port even that one needs some information over here you can see the pots that are available in my cluster you can have a look at my yeah the device notes I have on my clothes so I think this is also already an issue they don't think it's nation to have this as a read-only port but if they don't think it's an issue then don't put it on the internet if you are owning a closer because I think this is valuable information at least the tokens the

namespaces it's very easy to communicate with the API if you know all these things and you can also find some parameters I can show you later so yesterday it was kinda rainy note though and I was saying okay I'm gonna find out if I can find or 2250 on showed up because when I gave this talk for the first time showed up didn't scan for this particular port so yesterday I found out that now they do and there's like 109,000 course open on that port they're not all Cuba needs but you can do some magic by searching in a common name if you have a with containing cubes or node it's a good chance you'll find a Cuban nice

cluster and I found a Cuban is Buster and because I didn't want to go full blackhead I just decided to dump the logging of one port in a Chinese cluster because I love Chinese clusters and yeah what's not indicated you can read all the things there and this is another example of another Chinese cluster where you actually can see yeah you can see everything and all these spots you can you can own easily you can just spawn a container or you can even find a previous container and put your Manero manner on a Chinese cluster I mean it would be funny right a westerner a king a Chinese cluster putting on my new outliner there I think that's that's

pretty cool and also in bug bounties it's quite common nowadays to have a Cuban Easter cluster and I think that's ah their bug bounty involved someone hacking that cluster and the shopify one was a very interesting one there was a server-side request forgery simplify you can make your own you can make your own website and basically a Cuban a spot and we chop if I there was a method of making a picture a PNG file of your particular shop and some guy found out that there was a serious side request forgery in the tool so he managed to dump the private keys for the Natesville and using that he managed to I think yeah the metadata was concealed

I think in the Google Cloud you can choose to have the metadata concealed and I think you should because this guy managed to get the client certificate the private key and he started listing the pots in the Shopify environment and yeah just this poster because it was so it was censored and then he used to get secret command to get the JWT token for his name space service account and once it's all set for it is very nice and when he had that program he could just you know do an exact on any part and he was root on all instances of Shopify and this is really cool imagine having this in your bug bounty

and yeah it's impressive and he got twenty four point five K dollars I think that's a lot of money and a lot of Shopify swag I think he should only do it for the Shopify swag who cares about money I know he discussed if he has any so yeah what can you do I think I went really fast just like 22 minutes in my talk and I'm already in the end so with Cuba needs you can define separate tears you can have a fun time tear you can have a back end here you can even make an engine next year and your front end should only talk to the back end and maybe you know to some

other things but the back end should definitely only talk to the front end or vice versa so you can make some nice you guys filters there and again all the configuration is done in llamo so if you don't like your mold and Cuba needs is not for you yeah you can set up namespaces make nice administrative boundaries and you can also filter your service port on the outside so like for 10250 ten thousand twenty fifty five those are all ports you shouldn't do it yet those are the note controller for the cupid api SCV even the Calico network controller don't put that stuff on the internet and if you're in an internal pen test you should scan

definitely for those ports because usually the cuban is container the Cuban is cluster is let me see if you get access to a Cuba's cluster you can probably do some lateral movement from that house yeah and again do not expose the note to the internet if it's not necessary at least use a VPN or an engine X reverse proxy for authentication this is an example of method policy this is denying all you guys I think except for 53 a UDP and TCP people think that this is only for DNS but of course you can put a shell on any port but this is an example of you guys filtering it's also recommended to limit the

service exposure also don't allow everyone to SSH your cluster maybe use the bus your hosts with some whiteness in there and it's possible disable privilege mode yeah optimal service accounts if it's not necessary please don't do it and there are now some tools available like cube bench and cube Seck which actually do some audit the auditing of your cuban is closer so you know what to disable and what to have enabled usually it's just all what yeah it should disable in Cuba needs you can make a full security policy in any port security you can say you can only run as this ID a UID by default I'll post one with UID is zero so that's good but you can also

say use like any number like 5,000 4,000 then it's not good on your closet you can also set the deny access clearing exact thingy which means that you're not allowed to do a privilege escalation from the pod and oh no way the died the night isolating exact is that you cannot do an exact - I on a pot that's privileged so let's say you have one pot in your entire cluster that function privilege mode you don't want people to easily compromise that one you set the deny escalating exact so that way I cannot use that command pill on the pot and allow previous escalation mode which sounds very interesting as a security guy you should always set the

false because I think that's this big win yeah check your containers from vulnerable software because if people seem to forget that also containers can be vulnerable they're basically Apache web servers or mail servers whatever make sure you have a nice way of keeping them up to date maybe have something we used there and this is an example of a hardened port configuration I haven't made it run with UID thousand that has a GID mm yeah I gave it a nice name I think I copy most of it from our website you can mount like a volume there you said enough escalate into Falls the read-only file system to true because the root filesystem yeah there's no reason for

you to modify it run as non-root I also said to true because the I think both should not run as root and you can also even wipe this or blacklist some some Cisco so for this I didn't allow my pot to have wall sockets or I didn't allow my my pot to use the Chum command or the choice Cisco so you can basically make a very nice very nice like this there or things they cannot do maybe they can only a less or maybe they can only open a socket and almost socket you can you can wipe this everything yeah again service account should be limited apply to the principle of least privilege yeah nowadays she means people do this

so when you deploy deploy your cluster you will find that you don't have a lot of lot of things you can do but yeah keep it that way you know I mean it's it's hard for administrative sometimes to administer Acosta but when they want to make things easier which a lot of people now do for their cluster you can you can easily get yeah compromise the question yeah that's it that's my that's my tog you got some time left do you guys have any questions no questions you have a question yeah no I didn't but it's 70 possible yeah it's supportive actually huh yeah you know this question was did you actually play around with the sed nuts profiles on the

ports and I didn't but I think it's a good advice to do you get as you can never never look at that yeah and the other questions yes what do you think the likelihood is that traumas reused in the venous machine I think it's very unlikely I think he likes Windows 98 those three good times anymore mm related questions any more questions I just go over there yeah what do you think is the best way to midnight the detective kinds of this kind this kind of VA here yeah you can with login definitely you know there are some low clouds in the Kiwis where you can all see all kinds of this you can see this

behavior you can see the exact comments that are getting executed the exact API calls that are made so yes probably actually my next research is how to to do forensics on the console and how to how to create a dashboard for these kind of things so next year I'll come back to you with any more questions ah nice safe questions get like 22 or 32 more video manage to analyse manage orphans open-ended like for Mizzou everywhere I look at them yeah are any one of them more secure than by default it's it's interesting yeah I would go to buck bounties and I never look at them I think that's yeah but yeah no I don't

know no I think it's got it kind of depends you can for Shopify prisons they use the Google one but they still have a lot of settings they need to put themselves like they forgot to conceal metadata in this case so yeah I think it really depends on how you use your cloud instance but the I'll come back to you next time we'll answer a question any more questions okay then that's it if you have any questions I'll be walking around and thank you