Argghh, yer kubernetes be now a shark bait!

I'd like to introduce Alex if Caen can be a very provocative conversation about shark bait I like that just a quick introduction about Alex he's the director of solutions for Eclipse iam in Portland and he specializes in security solution architecture advisory and implementation of firmware and application security container orchestration and I am he's presented in many at many conferences and security events and has co-authored the aisaka CSX professional certification and when he has some free time he loves to climb mountains so please welcome Alex thank you everybody thanks for coming as if you've ever seen any kubernetes presentations they're always nautically themed and that's because kubernetes translates from Greek to Pilate or you know ships pilots so

we're gonna do it but we're gonna do it in a slightly different fashion you know we're but I got the guns Mike my boys were a little protesting but they all right okay good so we go alright I'm Alex if King you heard all about this we're based on Burton now kubernetes I want to take the first step in adopting the developer mindset because developers developers developers developers and this is where it all started so when the cloud came about the cloud made a great promise the cloud told developers hey you don't have to install your own servers you don't have to provide your own network infrastructure you don't have to provide your own storage we're gonna take care

for you and the developers said cool sounds great the cloud take off then the containers happened and container said look we're gonna package up all your dependencies for you and in fact we're just gonna deliver this whole thing for you and developers said awesome I love it 9k kubernetes so kubernetes great promise to the developers is the fact that you can pack a job all the dependencies all the infrastructure all the requirements for your application all you need to worry about is just getting application running you provide what's called a declarative model so you you just tell what you want and then kubernetes gives you all of that and developers set out some me and we love this stuff and so

it's said in the declarative model we're gonna move from this to this this is cool that looks great that's just awesome everybody took on this and thought hey no finally we can stop worrying about this you know pesky IT stuff and having to configure everything in in deploy it in and get it all deployed the way we want so then then the reality started heating a little bit and they thought okay so I want to throw a bunch of containers on the cloud and what I do I install the containers all right well then I need something to really to have first and the ingress point meaning somebody needs to comment to my containers and then containers

need to talk to each other when they're microservices they need to really communicate know how to find out about each other know what API is to expose so I need an ingress point and then man I'd love to monitor my containers too cuz it wouldn't it be nice to actually know if it's up or down and if there are any errors or logs and they said okay well we're gonna introduce the metric store and the log store and a couple of you know systems to monitor those stuff and then they heard about the security thing it's like yeah I just want to get them off my back so I'm gonna throw in some off server just just for fun and then

network controller all right well we're gonna do it and then they finally realize okay so now we've got a lot of other stuff that's in kubernetes we need to somehow manage that too and first of all you need to you need to provide a store where olders containers are coming from so you need to supply a registry okay we're gonna do that in order to install my stuff through declarative fashion or through an actual support method of package deployment I'm gonna introduce a package manager if you're familiar with apt-get that's essentially what it is except it's for kubernetes I also need to manage my security now with TLS security so I need something that would negotiate and prick

here all the certificates and install it for my containers and I need some storage management now of course I need a dashboard for all that so what you're looking here is what a real production level kubernetes deployment actually looks like it's not containers it's actually all the stuff that's around it and this is what we are gonna be talking about in this talk because when you look at the actual details of the Carini's deployment this is what's provided by the kubernetes ecosystem if you like a lot of components that make it actually work in production are listed here the ingress points I've highlighted some of the ones we're gonna be looking closer into ingress points the very popular one

is the sto it provides a way of connecting into your kubernetes cluster the package manager by far the most popular is home and tiller and you'll see it in the demo the service meshes envoy is the one that's most popular metrics collector Matthias is the thing that actually collects data from lost those containers and details and puts that in a dashboard a UI called Ravana and then you have network controllers with flannel and calico and TLS Meadows resource manager being the one that's most popular once and then you've got an image store which by far is overtaking by docker registry but there are also cloud solutions authorization authentications been having a slow pickup but it's out there you you will

hear if you're working in this area here about spire and open more are back manager's been awhile but what it is is a list of systems a list of full-blown packages that somebody developed that the whole soul need for existence is to support the infrastructure that you're putting in kubernetes so first of all how does it actually work and you need to understand it to be able to hack it the way that those containers are those supporting infrastructure is installed is called sidecar containers the sidecar containers is essentially another container that is running in parallel to your container you don't really have to do much except know where to put your containers on what nodes to deploy it on

and then II steel for example will take care of site loading all that stuff that it needs to communicate so nothing is in your application which is great but then there's a bunch of stuff parallel to your application that needs to all this instrumentation and that's a sidebar now an interesting concept that they came up with is called a mutating web hoax when I first saw that and when I heard about mutating webhooks that that picture can jury out in my mind so they're there it's a technology to be able to understand what you're deploying in a node so that an East EO or an envoy can actually figure out what it needs to deploy in parallel to

support the application that you're running a lot of complex stuff well even developers looked at it and said me and that it's getting really complex and then I heard you know all those cloud providers in the past did a great work for me in in getting rid of that IT issue I had can I have the cloud and eat it too and the cloud provider said absolutely just pass you know but 400 bucks a month and won't give you kubernetes and this is what they did so they came up and they started offering and they said okay so kubernetes has this concept of the control plane and this is where you basically run everything through where

your API resides where your database called at CDU resides where all the scheduling that does the the hard solutions for what's called a bin packing problem basically figuring out you've got this huge container where do I put it and because I've got a bunch of nodes and it sits on this one but it doesn't fit with this one but there's another plot on this node so it solves that issue schedule would help then they also give you a load balance tree to provide all the capability for providing high throughput loads into your containers and then they obviously give you DNS management and they almost throw it for free they also give you a firewall so you can protect your nodes

and obviously they give you notes notes VMs that's what they started with and they give you notes pre-configured for your communities they said hey hey mr. developer we give you all that and we're actually gonna do even better what we're gonna do for you is we're gonna take this the part on the right hand side and take it and maintain and manage it all for you you don't even have access to this all you have is an API that we give to you and it's a kubernetes api you can run all kind of calls against it using the korean alias itself and then we're gonna put everything else and we put our awesome security model on top of

it we're gonna put our you know provide external eyepiece and we're gonna have metadata and metadata sounds really juicy if you've you've worked with clouds and then all you need mr. developer is to bring this stuff in you bring your registry with all your containers you bring your rules for deploying and basically defining hey declare this solution and then we're gonna deploy it for you and then you bring all your secrets all your configs and cetera to make your solution work and of course you come with control tools in the dashboard so sounds really good sounds attractive this is why I took off and it's taking off even faster and faster the trick here from a

security standpoint is look I painted in two different colors and you have a fairly okay cloud security model on the green and then you have a you know it's an okay doing security model in kubernetes the problem is that they choose completely incompatible security models so this is what we're gonna look at and first I'm gonna actually lead with this slide so I did an analysis of different ways of deploying kubernetes and I'll explain it to you in specifics and so on a darker de this is a containment model that your container runs and when you run it on your personal laptop or or a server so it has an armor profile it has a second profile

and it's filtering syscalls I'm not gonna go into too many details on this but the moistest cost you filter the better you are essentially there are some dangerous syscalls that actually would allow you to abuse your kernel from containers so Daugherty counts with really decent defaults now mini cube which is a way that you can get communities started really fast on your own personal laptop or your server counts with horrible defaults you can see it's all real it's not confined it's disabled that it's almost not filtering any syscalls so that means you can you can screw up your own machine but whatever you know your developer you do to daily k3s is a compromise so Keith es is what they call

themselves as kubernetes - 5 and they tried to get rid of a lot of components and said here we're gonna give you crew natus that's easy to install just give us a couple of DMS and you're gonna have your own kubernetes server sounds great and it even has some app armour installed but it's still not filtering stuff and what I figured out is that if you have k3 yes if you don't know what you're doing escaping containers really really easy and it really literally takes one call to get if you're inside a container and you're running as non-root you literally just run on share - are in your route that's that's all you need and that's

because they are not filtering the unsure syscall so that's bad now cloud providers now those guys actually did a little better GCP is obviously I have it back that kubernetes came from GCP so they've been studying and learning it for a long time you see I've labeled those the three are in the yellow and that's because even though they're not filtering well what they did is they actually hardened the nodes themselves so the VMS that your kubernetes comes with has really really tight controls and this is good so even though kubernetes here is kind of slightly configured you can't really use it because the node an underlying VM protects you from doing that good now that that was fun and we're

gonna adopt a new mindset I'm gonna switch into pillage per jar all right so I'm gonna switch into a demo i i've redacted some of the IPS to protect the innocence but we're gonna do this so we're gonna first run this and see what what it shows and it comes up with a server eastern boy this is my first indication i know i'm hitting a kubernetes cluster then i'm gonna run and i'm i'll have to say I'm lazy I don't do a lot of stuff with curl when I have tools and so this is this awesome tool that says which cloud I run it it tells me digitalocean nice I know Digital oceans is fairly fresh I

have nothing against those guys but we're gonna we're gonna see what we can do in that cluster so this is an important step here I'm dropping into a container so I had a remote shell running and then I was able to execute it into the into the vulnerable application get a code exact and get it back to my shell so this shell that popped up is the shell that that's running inside of the cluster container you see this weird number in the D in the name of the host that that means it's a pod within communities all right Who am I and this is important to I'm I'm nobody really I I'm not our route and running in some

random container doesn't you know really explain much to me but I can write to temp yes everybody does and so what I do is I say okay I'm gonna first test if I have tiller tiller is is a part of helm which is a package installer until air runs inside of kubernetes helm is on the outside connecting to tiller tiller is always inside that's how people install stuff into kubernetes I know there's tiller and this is the default name for tiller so I know I can do some interesting stuff with it what I do is I just get myself helm client component of tiller and then I do this eight can I run a version on tiller and Helen tells

it yeah you can this is how I know I can talk to it and once I know I talk to it I do hey can you please tell me all the applications that you have installed and tiller says sure why not here have a look and this is what I do look at all this stuff so this telemetry stuff looks like what they installed in their application and then I can look at various other components and I I can see the hider is installed which is the authentication but you know I listed their stuff so what I do with that list well the next thing that I need to do is to ask for the metadata

hey dear digitalocean can you tell something about this node and they say sure there you go here's the stuff for this node now the thing to notice here is this it's the the bootstrap token and I'll mention it rather briefly but for a node to join into the kubernetes cluster it has to go through our authentication model that's how kubernetes knows that you've added a node to the cluster and for it to do this it shares this token essentially basically says when I'm booting I providing a token the token is exchanged for an SSL certificate than exchange for a different nacelle certificate and this is how the node is registered and I can do all that myself I know the token I

know how the node has been registered I can basically impersonate that node at this point through the metadata and of course I said I'm lazy so I'm not gonna do it manually I'm gonna run this tool cube let me in awesome tool and what I do with this tool is I first I do a bootstrap which knows about do metadata grabs the token exchanges for SSL Certificates builds the profile and then I tell hey I know where I need to go to talk to the API that digitalocean provided remember and then I ate it and run it on a note so at this point I am registered I'm completely configured and I can run cube CTL which is the way of

controlling container and I run this cube CTL I can get the secrets but I'm not really able to get the secrets because I am a node I'm not actually a user within that node and then what I do I I can run another tool I built this chart chart is the way of installing stuff into Hjelm containers and that chart basically says hey just give me everything just do that for me until it says okay so next time I run this the gets secrets I get the secrets and from that point on I can do anything I know the secrets of that cluster and one of the things I'm gonna run fairly fast I'm gonna use another tool called rag and it

allows me to list everything that's in the registry so I get it it's pretty awesome to build by just result run this tool pass all the passwords and URLs and everything I need I got from the secret and I got a list of all their containers all the images at that point in time what I do is I use another tool called darker-skinned built by a couple of Spanish guys darker scans has this interesting thing image modified trojan eyes I tell it what I want to Rajan eyes I tell it to inject a shell into the image and it says okay I tell what IP I need to reach out this is the shell that I'm

listening in and then it's gonna create a file for me and then after I've injected I need to push that back into the registry so at that point in time that image is replaced with a malicious image that I completely trojan eyes anytime anybody runs that image they're gonna pop a shell on on can reconnect back to me so I've essentially what I did is I not only persisted I'm also pivoted because anybody connecting to that or as you see pulling that image from that registry will already get my show so pretty good stuff and then yeah and then for the final part I can also shut down the cluster so there fern free love I'd like

to make a point that it's not very typical but it happens all right I'm gonna run quickly through the last slides because I have the five minutes warning assume darker has good defaults kubernetes in the France horrible defaults I'm not gonna go into a lot of details on this slide but you can read it later it's basically the certain manager has validation disabled East do have validation Tilly has bad defaults one thing I didn't show to you is the fact that once you have those secrets and you have SSL through less encrypt if you know how it works it actually one of the ways is that crates are record in DNS to create a record in DNS you need

shared credentials there in kubernetes so by me popping the secrets from kubernetes I have access to the DNS I can create a records only DNS so bad couple of words of advice helm is very challenging it suffers from an issue of nobody you validates the charts that it installs and it can install from anywhere like you saw I just download my own chart installed my own chart the worst thing there was tiller tiller has no reason to exist and in fact the are killing tiller finally next variation the felons coming called tiller less but the reality there's no there is no reason to have a tiller on a ship right so if you're if you're getting into communities worried

over eyes look for something for the next version of film register you saw how I owned the registry there's a thing called notary and it's based basically the signature so it's picky I signature is on your containers so that I can't might've if even if I modify the container like I did when you pull it it all failed because it's not signed properly now it's based on a platform called tough tu F but everybody who knows PGI knows PK is tough so when I did a research on over 25,000 docker images in docker hub that's the only percentage of those that actually had the signatures enabled so it's tough all right it's not to some people in the

room metadata is bad so hide your metadata GCP actually did a decent work they they if you use Google Cloud they do hide your metadata so that while Union your node needs to be boot strapped into kubernetes your pods that are running on that node has no business knowing any other metadata they should not care ever and that's it that's my last slide you can read it it's really about just trying to fail gracefully and making sure that losing a cloth container doesn't mean losing the whole cluster yeah [Applause]