Rory McCune - Kubernetes, Container Clustering Chaos

afternoon everyone welcome along to talk on kubernetes I thought I started off quickly by asking I'm guessing people have heard heard of kubernetes used it tried out a little play anyone using it in production last talk I gave there was someone that I said very bravely using in production now so yeah I came across kubernetes about 18 months ago some a consultant and I came across a new technology much the way I came across any new technology which was a customer said we want to talk to you about about kubernetes and seals looked through and they said you know what docker and this is kinda like docker so are you going to talk to which point

was okay I'd best learn about this then and over the last 18 months I've been trying to learn more about it and I picked up various things which hopefully very interesting and to help you work more about it you'll notice a lot of kind of sea based imagery if you're wondering why the darker and kubernetes communities love their nautical metaphors kubernetes is Greek for steersman so there's all about the kind of slide forget opportunity of some nice pictures very brief about me I've been in information IT security known for a fair number of years I'm a managing consultant at NCC group I'll also contributor a security Stack Exchange they don't ever heard of stock exchange gifts exchanger so it's super

useful if you like security grants or like stack overflow for security so you ever have security questions and answers feel free go there very handy I'm also contributing author at the CI a stalker and kubernetes standards so one of the things I found out very early on I'll start looking at this was those really know good security information about curators at all when I started looking this as a sort of middle of last year so luckily a guy started a standard on this the CIA I kept in and helped a bit a bit with that so we at least got something to start with it's what we're going to talk about what's the plan first things

were talked about is Cuba native the architecture so what is this thing how does it work I also found when I'm securing something to try to break into something is a little easier if I understand how it works so that's the idea start off with headers this work we're talking about with deployment options again one of the odd things I found about is there's so many different ways of deploying this stuff you can deploy your on-premises you can store in a cloud there's and lots of people who either sell you or give you an open-source project for how to deploy this says what happened please let out we were talked about threat model threat models really

important right if no one tax buying security doesn't matter how bad my security is if no one's going to attack me so what here who is going to attack me and how they may be going to do that it's really important and then we were talking about security concerns I hope to fix this stuff that's about her to break it then we'll talk about how to fix it because obviously we need to fix this stuff afterwards kind of Tillman's broke it go away right so what is it point to start off with you go to the website this is what they'll tell you it is an open source system for automating deploying skilling I'm managing

containerize applications it groups containers that make up an application into logical units for easy management and discovery and oh isn't that a mouthful basically to me what this is is a way of allowing developers to ruler applications quickly and easily in a containerized clustered scalable way so I want to roll something where I can bring it I can make it bigger quickly I can rule out new versions quickly I can handle my cluster nodes dying on me unexpectedly and this system should manage all the complexity for me I don't need to know anything about that I just deploy the application and it handles the complexity and that's the theory what about it will it come from

this came out at Google in 2014 some Google and Google had an internal project called Borg and some Google engineers thought we'd like to have an open-source version of this thing so they came up with cuber Nettie's it doesn't share any code with Borg but it's based on similar concepts it's managed by thing called the cloud native computing foundation that is he can have a conglomeration they were pretty much all the big players in cloud so Amazon Google Microsoft Red Hat IBM you know you name it they're probably involved in CN CF and this is their flagship project kubernetes itself where its first public release in 2015 that was at one point also we're not talking like really old

code here and it's got a really rapid development new major release every three months or so one of the problems I've had was securing and talked about security on it as they change everything so you see with my recommendation is this then three months later your recommendation changes because there's a new version and also really rapid adoption this is one of things that surprised me when I started looking at this so I started looking at it but I see not that long ago 18 months and they're already bank talking about using it so the UK is a bank called and they were tweeting about how they had ruled all their production infrastructure onto kubernetes last year

the UK home office or another big user they've got a lot of conferences and said they make produce in production of kubernetes and also people at Walmart and Ricardo they've been various pressure it is Lisa's saying they're planning to rule out new systems on to kubernetes so this is something that we're starting to see quite big adoption of people are making use of this stuff in the real world it's not just like some sort of thinking this is really cool because he real companies real money being deployed on this stuff how does it work let's start off with a demo because why not what could possibly go wrong okay so in kubernetes land everything is Y Amal so yet another

markup language if you've not come across it before basically the idea is you get resources so this is an application and this basically describes an application it describes what ports it uses it describes what is made up of and it basically has all you need to know about the application in this one big yeah more far and why should be able to do is at the moment if I do Cube CTO this is the command you use basically to kind of keep an eye on your cluster at the moment there is only one service which is the cuban IT service and that's the service that it's one's on every cluster but if i say cube CTO create

that's the young will father's looked out there it goes wait it says oh i'm gonna create a holder services for you i'm gonna create you a Redis master ready slave kind of front-end okay cool so now if I do the same command under before I say get services will you see that go hold of services I didn't have to do anything I've gone away and done that now it's put those on specific into specific care Tanner's I don't know what those containers are I can find out I shouldn't need to know this is just done it all for me and it's exposed the port for me three two three one six so now I should be able to go to here

and go to one which is the first note of my cluster and go to three two three one six right and that's my application so that's where about location deployed in what was that ten seconds and a laptop that's pretty cool and I can see test is very little silly a little guestbook thing but the interesting thing potentially is I go to another node in my cluster and I can see show me that and it shows me in my message from before and I can enter more messages so what this is done is this is actually what 202 route the traffic for me I didn't know how to route the traffic inside the cluster has done it all for

me so I don't chest submit all these messages so that I think you can see that to me is that's why a lot of is why it's interesting so why are people interested in this thing because it's really easy to do that and I get an entire complex application up and running with a single llamo file and some containers don't need that because the demo worked so all about terminology I've talked about this I mention these already is a lot of terminology in Cuba Nettie's land containers are docker containers so these essentially isolated up Linux applications all the container is really is a Linux application that gets bundled into a little package and gets isolated from the rest of the

system that's a container pods are closely grouped sets of containers so you might want to put your containers like really close together so they'll share things like the same network space and that's you put those together in a port services is what gets exposed to the outside world so pods all live in the cluster but the service is what the other side user sees and nodes it's just my notes in my cluster so the some terminology in there it's kind of wash the car mentioned pods lots because everything in kubernetes plan seems to be poets how does it work so it's not all that magic it's gone away it's created those things for me is created

the application but how did it actually do it what components did it make you sort of cuz if we're going to break into this thing we kind of want to know what we're gonna attack it looks a bit like this in the middle of the cluster you get this the API server this is the thing that you're interacting twist so when I sent that cube CTO command I sent that to the API server this is the absolute heart of the cluster everything else talks to that if you break into the API server officers gonna be really bad because you can tell the position in the middle there tells you it's a bit central but kubernetes is also it's not

our stateful service the API server doesn't store stay about how these things are deployed what it does is it puts it into a key value store at the top they use a product called etcd which is just a simple key value store and basically it kubernetes will talk to the key value store and it'll say store this for me get this back for me and there's a hierarchy of keys in there from a security standpoint again if I can get in there I can dump the entire cluster compact if I can dump the cluster convict I've got all the secrets that are in there I've got all the configurations I've got private keys or but also

fun stuff so it's bad and then down the bottom that's my master notes the API server lives there will be a number of worker nodes in my cluster I've got two worker nodes but you can have like 200 worker knows if you want each one of them has a cubelet on it and the cubelet is a process that basically talked to the api server and it'll say create a container for whatever image and that will then instruct the container engine which is usually darker so docker lives under all this stuff at the moment the tram probably going to change that but for the time being it's darker hue blue it will tell dock will say hey doctor

come get me this image from wherever docker repository I'm gonna up but the cubelet runs on every worker node and it has the authority to create new containers so again security standpoint if I can play with that I can muck about with the cluster so those are the key ones I kind of marked with a red net what King is one other thing to mention because networking to me when I started learning about this turns out to be kinda weird all my containers in there have their own flat network to talk to each other on you can have nodes in different data centers you can have nodes in different clothes but kubernetes will try and create a network

there is a flat network across all of them so that all the containers can talk to each other without needing to know anything about the underlying setup they don't need to know that they're in different places the weird part to me is cube really doesn't do this in in and of itself you put when you cluster the very first thing you'll have to do every single time is pick a networking plugin which are all third party plugins there's a number of them that kind of weird names like weave calico and flannel and there's a there's quite a few different ones but it doesn't do it itself so you have to manage the African or I pick one and go with it and there's

some security consequences to that but it was kind of odd to me because it gives you this big flat network and then people just say well you can just talk to any of the continued one that's great from a usability standpoint but I'm sure you can imagine from a security standpoint that may not be the best one of the things I've noticed what I'm doing container assessments as people might think that internal networks trusted now we all know about what assumptions of being or there's networks trusted get you get you compromised so yeah another thing I mentioned about deployment options before this is a little word cloud I did of all the different people who will give you a

different way of deploying kubernetes there's a guy who maintains a spreadsheet of than the 67 different methods on his spreadsheet and that's commercial products or open source projects there's things fool platform-as-a-service from open shifts read how open chef which is they'll do a lot stuff for you d like to layer hold or their own products on top of kubernetes all the way down to things are basically bash scripts they just you know around the components and configure them so there's a huge variety of options and what i've been looking at this the thing I found the most important thing is the importance is the importance of secure defaults because what I've found is that each one of

these options and literally these deployment options makes its own choices about how to secure the cluster what option to turn on watch and not turn on and some of them make what I would call some really surprising choices about what not to bother securing so I've looked at various ones and I filed bug saying did you really mean you know configure that in that fashion because it could lead you to this problem so all the problems I'm going to talk about as we go through this presentation I've seen these all in live default configurations of fairly large services this isn't just theoretical stuff I've actually said there's one I've got have a bug bounty poor bug reported within

the moment trying to get me to fix it because it's pretty much got all these problems but let's talk about threat model right so threat model who is going to attack my cluster how are they going to attack my cluster because that tells us what we need to do in terms of try to break into it and try to secure it we've got external attackers you do something on the internet someone's gonna try and break into it that's just the way it works so on the internet people will try and break into from that if you deploy kubernetes onto your internal one then anyone who can get on the internal one can try and attack your cluster right

they can get it if you can get to over the net one thing try and break into it pen testers will be trying to break into it pretty much guarantee attackers with access to a single container so if your attacker manages to break once either ten application is deployed in your cluster and you've got a number of different web apps and one of them got security bug in it an attacker might be able to compromise that one container so an attack scenario is how easy is it for me to go from compromising one container to compromising the whole cluster and this is the one where I've run into some problems when I've been discussing this

with kubernetes people because a lot of them don't see that as a valid threat they don't consider that as part of their threat model their view the world is if you the cluster is the security boundary once you're in the cluster we don't care personally I don't think that's a great idea from my experience of security that's not a great plan but that is what they do and that's one of the problems I've had in terms of get people to change their minds about how to deploy things and the other one is kubernetes allows you to have different users and you can have different applications running in a cluster so the the chance of a malicious

user turns up either a user loses their credentials or you get someone who wants to do something on a cluster they shouldn't be allowed to do but it's frustrated that they can't do it and so what could a bad user dude that's my other attack model so what we'll do is we'll go through each one of these let's talk about external attackers first I'm a tester the very first thing I do someone gives me a network service is that port scanner its first point what kind of what I got to play with here what have I got this potential and the kubernetes is great because you got lots of potential we've got 2379 which is etcd we talked

about before we've got 41 94 we've got this one here I kind of mentioned it is that people can't make up the main what port to put it on so it could be six eight six four four three eight four four three just four four three that's the API server the next one is a really good point it's not good if you see it because it's called the insecure API server that's actually the technical that's what they call it and cue branches line insecure API server and yet people will enable it sometimes 10 to 50 and ten to fifty five are the cubelet and we'll talk about more those and various flow Network plugins so network plugins like I said

you don't know which one you're gonna get they sometimes have their own ports some of them use BGP so one of them will deploy BGP on an internal court to manage the the networking of the cluster so that's got its own attacks reference obviously some of them will run their own etcd instances things like 4001 you'll see that but basically the yukio variety of stuff there so let's talk about however how we can break in some of those we'll start off with C advisor so C advisor is pretty much on every cluster by default I've seen it'll always see you there and if you busy there's an HTTP service it's unauthenticated indeed there is no

option to make it authenticated there's no flag you can set ventilation on to it and it's unencrypted so if you see this on a cluster you're going to get this it doesn't give you like complete compromise but it gives you some quite nice information if I'm a pen tester it gives me some interesting I'm at information if I go in here it will do things like it will tell me all the pods I've got running so that's all the different containers I've got running on my cluster that's kind of useful information it'll tell me what the rush to doctor I'm running well if it's really all rushing maybe I can find an exploit Hill tells me the kernel version it tells me

the operating system burst it tells me the directories things are deployed to so if I've got some means of executing code or pulling back contents of files I guess some directories and also down the bottom it tells me all the different versions of the different components I'm running so whilst that's not game over nor is it really great or or should that be exposed but that's pretty much on every bus driver looked at if you can get to the poor it's going to be unauthenticated and you can do that too that's kind of interesting don't worked okay attacking the cube low so the cube loads got two ports it's got read-only and it's got readwrite if we look at

read-only it's got various endpoints and one of them is slash pods so he only has a couple of things it's basically the idea of the read-only port is therefore statistic services lots of you wouldn't like a nice dashboard that says how your cluster is getting on so this thing is designed to provide the input to that it's unauthenticated by default and there is no third option to make it Ascenta cated and it's unencrypted so not great this again Chuck's out a load of information in a nice JSON format but the probably the most interesting part if you're attacking it is this what this particular node has got the API server running on it and what it does is it

tells you all the command-line options if we're thinking kubernetes LAN is a command-line option literally everything so if you get this you get all what sorts of fun stuff like what port we're running on what's the what's the key file for the private key for the PTI that's securing it tells you the names it's not going to get you to compromise but if I'm attacker and I'm a pen tester I'm a really happy pen tester right now because I've got this great starting point that tells me all this kind of authorization what am i using what's my certificate name oh that's sort of fun stuff and that that's fun but it's not it's not compromised fun if you get the readwrite

port then this is on SSL all that's self signed add our exception as we always do you can do things like dump out all the logs so that's kind of handy that lets me dump all the logs from the containers on the thing which is useful but again it's not compromised it's not what we want if you get the readwrite port so you get port 10 to 50 which is what this is on your probably and you can get to it with an authentication earlier versions so one things I said as curious keeps changing it you go by two I talked with earlier versions but I mean like six months ago I'm not talking like years and years ago

people are running this stuff earlier versions didn't have authentication for the readwrite port so if you could get to a network level you were going to be able to execute any commands you wanted newer versions do have that I still have seen clusters that expose it definitely internally if not externally what you can do here and I'm gonna copy paste this command because I can never get it right when I'm trying to pit here all these things are HTTP api's which is really useful because it means that you can just use color so if we clear so what God is a command here which basically says go and do a posts to the cubelet API for me

there's the poor and then basically here's the the port I need to post that to tell it to go to do for now I got that list so I got that from the read on read write ports I know I can easily find this stuff and then you do all the way through here and then at the end you tell it what command you wanted to run I'm acting anything you like frankly and so if I do this that file I'm trying to retrieve essentially the file that the administrator users to authenticate to the cluster and talk a little demos being happy yeah and there's my client key data there's my name I'm the Cuban

it's admin and everything else and you can do and you can also if you're interested you can do things like I'm so there so you're basically executing has a route inside a container at that point so if they haven't authenticated the cubelet API and you can get read right on it for each node that's running on that's game over that's I'm going to compromise every container running on this on this node it's not good news so that one's bad if you see that one or if your attacker is really good if you're a defender selim also good so cubelets are bad my demos don't need those let's talk about malicious containers right so I've been a pen tester for a long time it's

not uncommon that on a web application you may get some kind of minor command execution like you can get on today you can you can execute commands maybe you can upload a shell maybe you can get blamed command execution maybe you can get SS RF right so all these things are HTTP API so if I can make the server execute commands on my behalf what can I do to the API is there all these TTP so it gets you to create increased attack surface now I've got container so we're gonna have a guy who's got a container he's got that far but how can he get the whole cluster you've got access to the container file

system right so I've got access to the container that I've compromised fine that shouldn't be that much in there just some binaries and whatever it needed to run I've got that internal network position so now I'm on that internal cluster Network I'm like I said a lot of people who'd make these systems don't regard that as being part of their threat model so once you're in they've got less a lot less protection it's easier for you to get further on and obviously the way the weight internal kernels with Linux containers work you can attack a shared kernel all containers on a node one against one single Linux kernel if there's a kernel vulnerability you can try and use that

to escalate out to get access to Noda so let's talk about this was really fun a service account tokens are fun so in the kubernetes land every container when it's launched gets a series of actions applied to it by the API server basically a series of things it does to set the container up and one of these copies are service token into the containers with moments of file inside the container and the goal of that token is to allow the container to talk the API server so it says here isn't a thing for you little mister application you can now talk to the API server using this token until very recently those tokens were all cluster admin tokens so

basically if you could if you can execute against one you could do anything with the cost wrapping to do which is kind of not good as an example if I do Cube CTO create my say if I've got a container called bad container because you know all right so I've now created my little bad container this is just to simulate my compromised container and I do cube CT okay so that's me now I'm now in my I'm never inside the container network and all my running on my little bad container now I'd like to say that there was some super elite ninja thing I had to do at this point to get cluster admin but in

reality all I did was a downloaded the cube CTO binary and then I run it and it works and I'm no cluster wrapping I don't think I want and the cool thing about that is what it does is oops what it does is is kubernetes very handily when you start a container but other things it has it populates a whole lot of environment variables so that you know where to find things because obviously the plots can get started up and tearing down and all that sort of stuff so you can't reliably have IP addresses so it sets a hold of environment builds and one of them is it tells it we're twenty API server and

then cube CTL will always look for a token in the fixed location the valid there are this default location so basically as long as that tokens in place and you download keep CTL you get to be fast rapid which is fantastic if your attacker not so great if your defender this actually stung the b-side San Francisco they ran a CTF and they thought they'd be really cool mid one it insane kubernetes and what happened to them was the very one of the guys popped the first flag got sure on the first pip attack then hang on this is a kubernetes cluster got cube CTL thank you very much my cluster took her all the other

flights really easy CTF I so the guys gonna be said CDC I'll just go they didn't realize that interfere this isn't whilst if you talk to enough cab rights people they will this is no one it's not like a kind of super-secret nor is it massively well-publicized that that's the case so yeah and it lives it lives in var run secrets and basically that's the service okay so you just carry out the token there's just a text fault and if you present not talking to an API server it basically says yes you've got whatever rights in this case in this case you are cluster happen which is great service account tokens yeah dangerous things I'm kind of

Alyssa there's a default choice this really surprised me because I'm like this okay yeah you see why you might need it but not but to fall yeah epi server attacks before I mentioned there's obviously the insecure API in CI server I do know of at least one cloud service provider who runs Cuban auntie's who still has this available for the internal customer network so if you're in the customer Network this is available that's what is just as easy as compromise used to keep TTL - ass HTTP tell it where to go actually if I pull using the variable first yeah so same thing if you can get it into insecure API server it's pretty much game over

just tell cube CTL where to go just tell it what the endpoint is IP address port number and it knows the semantics and it want from there so if you get the interior if you guys ever that's game over to a CCD so mentioned etcd if you get access to each CD then obviously that's game over as well the cause it's got the full container config through cluster confidence got all the different things it's similar so I'm gonna come on for this one because I never remember this one so basically it used to be it was a nice HTTP API and you can use carol-anne all that stuff but they've gone to this binary format now which is

kind of annoying I'm still in the same iPad container what that's done is that just dumped that just that can I just says go and dump that that the the config go to a file so it's called test2 DB and then it's in this format called bolt DB which is no fun I've ever come across before but you just do Paul browser and then yeah that's that's the confidence right so as you can see this some kind of binary nonsense in there because it's serialized GRP see these days but there's enough text in there that you can you know either deserialize it if you really want everything or you can just get all the various keys and

tokens that aren't living in there yeah so if you can get a CD that's kind of interesting I'll talk about this because the authentication of for it's kind of interesting so I'll talk about that in a second but yeah attacking each CD is bad as well and again it's game over because it's the full cost of comfort and the OS kernel like I said I mean you can try and I won't go to my hungry no I'm just it depends on the kernel version you're running you see that's gonna be you can run something like a my contained this is a really useful tool if you're doing container testing basically is a one-liner and you just basically tells

you some stuff about how the container is configured and how what containment options you've got set so if you didn't contain your assessments and I containers are really handy you can do it all manually with Crockett if you like it but it tells your capability you've got various other bits and pieces whether user name spaces are in you so how easy that's gonna be depends on what person the colonel you're gonna be using but it's a kind of a cool technique anyway okay so talk with malicious users right so we've got a big cluster we might have lots of users and we might not want them all to be cluster admins so obviously we're gonna try and

restrict them and lock them down from doing stuff the problem is you have to get people if they're users keep CTL access basically because all interaction is done through that I saw another presentation and they were talking about the fact that they wanted to not give their developers SSH access into the cluster so said oh we'll just have them use cubes ETL well you've seen like to do cubes ETL exact it is SSH access it's everything access you can execute commands of cubes ETL you can spend up container to leak containers SSH into containers anything you want to do with containers cube details really great program but if you give people access to that they can be part of that and one of

the problems with some containers is there's an idea in container land of a thing called privileged if you've read any docker security guidelines it's basically the first thing it'll say do not allow privileged containers a privileged container is basically turn off all the security just turn it all off don't isolate me from the node I want access to the underlying stuff right now you can pass an option if it's not locked down called privileged is true and the other thing you can do is you can mint stuff in from the host if you want to by default and you can basically monster like mountain from the host I can mount in the root filesystem into a directory now you might not

supposed to be bad the interesting thing for me when I was looking at this is there's an option that you can sit on their API server which is a low privilege true or false every single cluster I've looked at is true and the reason for that is things like the network plugins need it so they default to allowing it which is a bit of a problem because it means you can do something well it like this so I just do cube CTO or create - a if for a quarter a crisper pod and then I introduce it so ask me created my privilege pod on Angel cube CTO

that's me inside my privilege pod and if I go into the node directory that's the root filesystem of the node right and if I go into TTC all the files are there and if I see Who am I they said that route so as far as that node is concerned I'm rude if I want to edit a file I'm editing at UID 0 I wrote it let me add a shadow' sshd config anything like it's game over for that host so if you can do privileged containers that's not good news privilege containers are very dangerous should not be allowed I max what ya access to nodes is really bad because basically the keys in there that allow

it to communicate each node to communicate with the cluster master so they're dangerous so key security considerations I've just shown you there's various ways that if it's configured wrong this stuff gets broken really easily how do you configure it so that doesn't happen turn off the insecure port clue is in the name there's no reason for loving it if you absolutely have to have it find it to localhost I've seen some people binded localhost which is less bad it's still not like completely safe but it's all about that allowing it to be to any other interface API server authentication everything should be authenticated kubernetes authentication is a bit weird it used to be they used HTTP basic auth

or token off and those are both static files stored on the api server if you wanted to and in the clear obviously the creds are all not clear which is not great if you wanted to change anything how to user a movie user you had to go in association to edit the file we save it and then reload the service that was the only ways changing user so that's not really a great idea most people will say don't do that anymore instead use client certificates so basically you get our certificate authority you can get with kubernetes and each user gets a client certificate created for them and so information about their identity gets encoded into

the certificate there is however one thing to watch with that which is there's no certificate revocation concept inside kubernetes right now so if your user loses their certificate you either reissue all the certificates or you wait for it to expire those are your only two choices there are external authentication options and things like webhook options so you can you can set up a web hook and do something else yourself but within kubernetes that's pretty much it certificate points to because it for the best of a bad bunch authorization so at the moment any older clusters they tend this just make you close driving so they just say if you're in your anymore open above the

authorization newer versions last there are four or five months you start to see are back come along role based authentication that's obviously a great improvement but you do have to make sure you apply it to give you a it can get a little bit complex and I give an example rules that's the basic cluster rules that come with the cluster what are you adding any so you definitely better complexity in there it's not like the super simplest thing in the world to manage the other thing that it's important to know that I think it cause people a lot of problems when they start rolling stuff in production is there's no concept in Cuban eighties of a user database

kubrick she doesn't know about isn't like store database of all the users what it basically does it relies on certificate that's presented to it to tell it what the identity of the user is so easy to keep it secure eighties admin give me a list of all your users Neil Goldman don't know don't me users huh it's wherever the CA administrator self control axis the Kuebler no event no honor no no animus off turn off the read-only poor these things are all options that's the good thing is they can all be turned off the good thing about it now is you pretty much can secure this stuff it but it six months ago these option some of them didn't

exist no they do turn it off turn off the advisor turn off and on us a tenth occasion it's definitely the way to go control access to each CD the interesting thing about eighty CD is the way it offender case is it basically you tell it when you launch it this is your trusted certificate authority and it will say any certificate from that certificate authority by trust and it will give full access because it's not they have an authorization concept when used with your days so if you can get a certificate for any certificate that was issued by that certificate authority you can get access T CD so if you only use one CA se you should all your users

client certificate so they can do their stuff and they can get a network level t CD they can point it their certificate eight CD and say hey I entrusted can either date mr. Kloster please and it'll give it them which is not great so that needs careful thought as to how you handle that other things think about um obviously allowing people to run privileged pods is very dangerous you can say security policy so you can basically say these are all the things that are allowed to be done you should only allow these and you should restrict users and what they do I don't think there's any good way to secure cluster without doing that so pod

security polls is kind of a requirement it's fairly new but its deficit since dip although maybe still and beat up but it's either be charged stable security context you can tell it you can low it lock down a container in terms of what it can actually do you can take him abilities offer you can take other rights off it you can apply a see limits policies to it in all she's done any limits policies knows that's not super simple but it's worth looking at if you were only seven production Network policy this is fairly new this came in at 1.7 in to release but this essentially solves the problem with the flat network so where before clusters

would tend to be flat networks it's now at least possible to say this container can only access these are the containers on these pores which is absolutely requirement I think if you're gonna have a secure cluster you basically have to use hot security policy and network policy otherwise it's very difficult a lot of the stuff that improperly resources a couple of resources there's a CIS guide C is very very very annoyingly recently went to this thing where you have to go to this page give them some information hit send and then keep the same browser open until they send you link and then put a link at the same browser I don't know why but it's

warning in advance because I find it really annoying when they started doing that but you can't get the benchmark it's free doesn't cost you anything apart my fake email address you make up and also I started doing a little tool to try to analyze some of the stuff so I actually don't know very briefly I did a he

so basically what that does is get all the CIS checks which are all the different command line switches and basically for each cluster it just says have you passed or how do you feel and what was the evidence for why I think that's the case and then the bottom I started doing some authentication some sort of vulnerability checks here so things like do you have external authenticated access Kuebler do you a low internal access to cubelet is the api port exposed all that sort of stuff just kind of basic checks and one of the problems I've run into is like I said the 67 different ways of deploying this it turns out is really hard to code

something that handles all 67 different methods so I've kind of covered the basic ones but if you use an obscure one this probably won't work brilliantly that said then once put any issues of pull request if you like writing Ruby and who doesn't then it's written in Ruby it really should be fairly easy to modify but yeah that's another resource conclusion so I think why it's about kubernetes is it's a really cool product and older versions of it were really not very secure at all newer versions you can secure them but they may not be secured by default so now it's at least possible to configure in a secure fashion but you're gonna have to some

effort into it in all likelihood the default security options I think we're super important multiple this is quite a complex product once you've got out running in production I don't think a lot people gonna make major changes to it so try and get things like quad security policy and network policy software before you go into production nor afterwards and always think about your threat model and attack surface so if you're worried about compromised containers make sure that the person you're getting software form is worried about compromised containers because if they're not you're gonna have a bad time questions

I was gonna say if you get anything from kubernetes 1.5 or earlier this don't have all the releases we get like one five or earlier it will just be vulnerable because they all were really awful apart from that I don't think so I've seen docker ones I haven't seen anything procuring a teacher too hard to do wait yeah but basically if you look for all the ones look for anything like curates 1.5 or earlier is likely to have most or all of these problems and even later once some have they don't I don't think I've seen no internal I've asked both as a project if everyone's doing one project kubernetes I'm not we're not more one yet for a learning system it's

not so far anyway the questions I do not have any I've not done any work with her office regarding their accumulated employments I would sincerely hope that the UK Home Office have had appropriate assessments of any things before they deploy to production but I'm not I don't have any kind of they don't whilst they will they've gotten public confidence and we use this stuff they haven't said that these are our conflicts so I thought the system will say look if you did or didn't do whatever I do hope that I don't lock them and I hope Bonzo who are a bank have using for a while I hope both of them have like thought through all this

stuff and like sorted all that I'm sure they would it was

yeah yeah if I was picking if you go in cloud-hosted either Google ghouls own version of it leches gke Google container engine is pretty good because people kind of wrote it the other one that's good is all shaft read how old chef so read how open shift is like that their powers and this system it's got a whole rest of the goals based on proprieties that's the one honestly we're seeing more of in terms of deployments and its best a lot of the security stuff in cuba Nettie's they have written it first into their own product and then they pushed it back upstream into kubernetes so if I was someone said make it made me pick I'd

say Red Hat boom yeah we go when I will pick