2017 - Kubernetes Container Clustering, Catastrophe? by Rory McCune

okay afternoon everyone welcome to the sleepy Gaul by buying after lunch time slot ice will try to keep people awake um so this talk today is about kubernetes you can see quite a lot of nautical imagery docker and kubernetes projects seem to love the analogies to do with like container shipping containers they've taken it really far so kubernetes is Greek for steersman so I thought well that's a good opportunity to find some nice pictures to put in the presentation just before I get started I'd guess if people heard it I guess he heard of kubernetes is a yes usually protectees like to fire either running it in production to do brave people running in production fantastic

excellent so hopefully some of this will be interesting and I'm what I'm going to do as I got involved in kubernetes about just over a year ago it was about middle of summer last year and it was how I usually get involved in new things which was client came along said you know about docker so people said oh well you can't talk to people of kubernetes that I said what so I did a better reading and try to work out what this kubernetes thing was and after a while I found some stuff which was quite interesting on it was interesting to me about how this this product was developed and that's where this presentation came out of

really it was from my explorations around kubernetes and things I've found that could be interesting if you're looking at it or things I've found that might be interesting if you want to break into it so if your security tester you might start running into this stuff on day to day basis and it's kind of there's some stuff to know but that's what this is about very brief about me I've been in security for a large number of years I'm a man to consult the NCC a contributor at security Stack Exchange people heard of security Stack Exchange so I've asked the question started yeah cool so if you haven't heard of it it's really useful it's like Stack Overflow

but for security questions and there's lots of helpful people there who are happy to answer questions it's a good place to go if you have questions I'm also a contributing author to the CIS docker Kibera aunty standards so there are a couple of standards one of the things I noticed when I started looking at kubernetes was there was very little documentation about security so luckily a guy started out a standard for it till in the last year and we've got that freshness at you up-to-date with the labor actually kubernetes net 1.7 so that's kind of worth looking at as well I'll talk a bit more about that as we go through what we're going to talk about

today start off with kubernetes architecture how does this thing actually work I'm gonna go to bed about that I always think it's really important you're getting to a new topic is you have to understand how it works before you can understand how to break it or it's a lot easier if you do understand how it works so what going to be a bit of high-level better talk about that we're going to talk about deployment options because one of the things I found out when I started looking at this was there isn't just one way to rule this out there's lots of different ways and it really matters when you're coming to look at it in terms of security how it's been ruled

out threat model if no one attacks my system it doesn't really matter how secure it is so who is going to attack a kubernetes cluster how might they want to attack it so we're gonna try and walk through some of the common attack scenarios to see have something might break into something and key security concerns no point talking about how to break into it I can't talk about how to fix the problems so we'll talk about some of the settings you should look at here deploying this to make it a bit more secure what is it first thing start off with so the kubernetes website will describe it as an open source system for automating deploying scaling management

of containerized applications it groups containers that make up an application it's logical units for easy management and discovery isn't that emmaus full basically it's a way of making letting people who have want to deploy an application and they want to deploy on a cluster of containers and they wanted to have a nice scalable system and they want to just work and they don't want to have to worry too much about how the underlying cluster does its job they want to just deal with our high-level say here is a system I want now to scale this I want to make it work I want to upgrade to a new version and I just wanted to work and the idea is

kubernetes kind of takes away a lot of those concerns in theory abstract them away and lets you as a developer that you're a developer system not have to worry too much about them it's last theory Fujii background just give me a bit where it's come from started by Google in 2014 so Google run everything they do in containers and they had a system internal system called Borg basically some people said that was really nice it was an open source outside version of this Borg thing so some people from Google starts with kubernetes in 2014 in 2015 that got its 1.0 release so the very first public version so it's pretty new we're talking like sub two years I think it is two

year anniversary the first public release just now and it's managed something called the cloud native computing foundation which you might well not have heard of but all the big cloud players are involved so Microsoft Google Amazon Red Hat IBM you name it if they're involved in cloud stuff that problem members of the CN CF and there's a whole world of projects of which kubernetes is probably the biggest that come under their banner and rapid deployment and rapid adoption it's fair to say that this is a very quick moving project basically new releases every three months which is a nightmare if you're a security person because it means things change every three months and something you can tell

someone give us some advice new version comes out certainly the advice doesn't work anymore because something's changed and rapid adoption is fair to say as well when I started looking at this I was quite surprised because companies from financial services so our bank called Monza they used kubernetes and they would be talking about how they've used it for quite some time and publicly so they're big use that's get use they're the UK Home Office our bigger users of kubernetes and they've done presentations publicly about how they use give or Nettie's so using the public sector it's also used in places like Walmart and Cod or both recently did announcements about how they were moving to kurban actus they were deploying a

lot there so we're seeing a lot of Industry adoption of this this isn't just something that you know some people are playing with we're seeing large companies really getting into this stuff so let's talk about the architecture let's talk about how it all works first however I'm sufficiently mad to find new demos so let's do some demos ok I have got a cluster I'll using this this command cube CTL is basically how you interact with clusters that curate these clusters pretty much at the moment I don't have any services so I just have to kubernetes service now that's the service that has to run to meet the cluster work so it's just sitting there what I can do is I can say

cube CTL create - eff example and it'll go away and it says here I'm going to create some stuff now well how did it do that what it did if I find example that yeah here we go basically everything in kubernetes land uses Yammer so what you do is you chuck Yambol files at the server you say I'm going to tell you how this application is defined in this llamó file you will go and create it for me and that why just chuck that it was this and this basically has a series of containers soon as that occasions and it defines how they work and how they interact with each other and know if I go here and I do cube GTL

get serviced again you see it the older services and I've got a port here of their 306 five two so if I go in here mm-hmm and here and I do got my cure Bonetti is one and I do what was that Court 306 where do we get a guestbook that's my applications I was run up two instances of run end and some Redis slaves and masters from that yellow far and I can say there's a message and it's a message and then the interesting thing is if I go to another node in my cluster you know to it exists there too so it's exposed and it's worked out how to root and it's routing that traffic into my

cluster to one of the application nodes and it's doing all that for me I don't actually need to know I don't need to know where those containers are running I don't need to know how they work I just need to tell it to do it stuff I might say hello and it does it so it works quite nicely so that's you can see why people like like that right if I'm a developer and I want to deploy something and I can say to an ops guy or I could say dev ops tile I'm just gonna deploy this and you pass it to that there you go great excellent works really nicely as did my demos which is always nice right

so I don't need that because that's a video feel it didn't so a little bit about some terminology quickly it's worth kind of covering off scurvy it started using terminology already and it's worth mentioning what they are containers if you've not really come across them it's basically a packaging mechanism for an application that gives you a nice little way to package up an application in a standard format docker images are the default standard there are others however most people they talk about containers I'm talking about docker images a pod humanities has the idea of a pod and a pod is a group of containers that work really closely together so they share a network namespace if you deploy multiple

containers inside a pod they get the same network stack they get the same various other namespaces as well so they're very tightly put together a service is what gets exposed so that I created three services there a Redis service back-end and front-end and a front-end web service so basically group the pods together and cubed then says the service gets a stable IP address the pods they could be created they could be deleted they could be moved you don't need to know any of those things you talk to the service the service stays put so that's an important one and nodes are just notes in my cluster so you got a number of notes to make up a cluster you can you've got

kind of masters and slaves on top of that I said so how does it'll do it right one this very cool thing it all just worked do we need to know how it works well this is how it works so basically the API server right in the middle the API server right in the middle that is the sort the beating heart the cluster if you look at all the lines on the diagram there they all go through the API server everything talks to the API server it doesn't talk directly component to component they all go through the API server so from security standpoint the API server is obviously our key goal right if we can

control the API server we're going to compromise the cluster because that's where all the stuff goes through however people in these clusters are stateless so the API server doesn't store its own state it doesn't know anything about how pods are put together it stores it all in etcd so at the top that's a key value store and you generally get a cluster of those so you'll have a cluster of etcd stores and what they do is they essentially store all the configuration in the cluster so from a security standpoint if I can get to that that's bad too because I can drag the entire configuration and I've got all the secrets and everything else that goes

into your cluster and then down the bottom there so we've got that's our control plane those three components there then the mote we've got our worker nodes so they're the ones actually run the application and the probably most interesting component there is the cue blue and the Quba essentially talks to the API server and the API server will say go and create me some containers go and create this use this image uses docker image run it up expose the service do that do this and the cubelet handles that on each worker node so obviously if you control the cue blur you can run containers which again is going to be bad for a security standpoint so those are our three main

ones and then the container engine over there on the right is generally darker so docker sits underneath all of this you don't have to use docker but most deployments use docker so that's how it all hangs together one other thing to mention it but this is networking because kubernetes networking is kinda weird um when I roll out those containers inside that application they all get to talk to each other across a flat network no matter what underlying node they are on they can all see each other and they have their own addressable network and is a flat network by default and it doesn't have any firewall by default so typically anything on the cluster could

see anything else on the cluster which has obviously security concerns because some sessoms I've done a bunch of assessments of container networks and people might make this stick mistake of assuming that that container networks trusted so they'll put things like MongoDB instances with no passwords in the container Network assuming you can't get to them turns out not to be true in some cases the other weird thing about it is kubernetes doesn't do its own networking it assumes for some reason you want to use a plugin for that so it passes it off to third parties and you'll see things like we've calico flannel ramana lots of projects with Kenna weird names and they provide various different ways

of doing networking but the advantage is clear because it means I could have a cluster of running on different cloud providers at different nodes in different clouds different networks as far as the apps are concerned they're on the same addressable network very important in terms of making the thing work oh and deployments yeah I mentioned this before this is the word cloud I made of all the different deployment methods I could find if there's a spreadsheets gets maintained there are 66 different either corporate supported or open-source projects which tell you how we give you ways of deploying kubernetes clusters um the very firm just like bash scripts that deploy the components all the way up to platform as-a-service to suddenly

read how OpenShift but it's a whole lot of stuff on top of Kibera at ease um the thing about security that is they all have their own idea but what secure means and they don't all have their own idea about what default security option should be in place so if you're assessing a cluster or thinking about deploying it probably the most important thing is what are the defaults that my cluster deployment method chose what did they think I had to do from retirement security because once it's up and running and you got production workloads on it let's be honest a lot this stuff's really hard to change so what secure defaults you choose it's probably the most important decision

you're going to make when you're rolling this stuff out and if it's the one thing you think about take away from this presentation is that's really important because I did quite a lot of what I was looking at this I got various default install mechanisms for open source projects and from cloud mechanisms I installed them and I got some interesting surprises about what some people think secure default should look like major providers think their default security option should look like definitely not what I would choose a secure so let's talk about some of those model threat model super important if you never get attacked doesn't matter if you're secure all right okay for what security in the

world if no one ever tries to attack you who cares but there's obviously this is the real world and systems always get attacked who's going to attack Acosta external attackers right if you deploy your cluster on the cloud then it's gonna have some internet accessible component so someone's going to try and attack it if you just will deploy your cluster in the corporate network you don't know one people can attack systems on a one external attackers are factor like this one is a wee bit more interesting what about an attacker who's got access to one container so you've ruled out an application at your container it's got a security flaw the attacker gets access to that one container how hard or easy

should it be for them to get cluster admin once it done that this one is weird because this is the one I have a difference of opinion to some people who make human it's deployments some people don't think that's part of our threat model so what I say that they will deploy clusters they'll say the default option basically if someone gets access to one container cluster app and should be trivial and is trivial like really easy we'll see that and that's where I have a difference between that's why I'm saying be really careful when you're choosing your cloud to plan your choose newcomers deployment method make sure that if you think that's a problem that

they think that's probably and the last one is what about an attacker who's got rights to run a container so if you work in a really big company and you start rolling this out across your corporate you might have groups of people who you want to be able to use it but you don't want to give them cluster admin all right you don't necessarily want everyone people to get full control of your cluster so how easy or hard is that going to be external attackers so start from the other side world right we're gonna start as a point of view of someone who's trying to break into this cluster from external so what do you do

as soon as you try and test assess the security of a network system you port scanner well it's that's what I do and if you port scanner kubernetes cluster you get lots of potentially interesting things so we've got 2379 that's you mention event ID etcd so that's the default port for the key value store if you see that that means they've got a clustered version of that running so they're listing on a network interface and you can try and attack it we've got 41 94 to see advisor the one below that which I've got kind of in brackets it's an eight it's because they can't make up their main what port to run that on some run it on six four four

three eight four four three or just four four three that's the API server everyone knows that's gonna let the most important one the next one's kind of interesting 8080 is the insecure API server that's a call and you can guess from that that it may not have the best security in the world and I have seen that exposed inside a cluster by default so anyone in the cluster can just go 8080 thank you very much I will show you that and I set ten to fifteen to fifty five or think what are the pupils we talked about about that and then various so that depending on what network plug-in you go for you're gonna get

various ports there some of them do for example use BGP so you'll see the BGP ports often you want to what he'll this is an internal system they use BGP to manage routing so that's interesting in and of itself some of them will have their own etcd instances for managing their configuration so you'll see another port VT CD they don't like 4001 or somebody so let's let's let's do some more demos because you know why not first we're not see you've either got it's loaded so see advisor is installed by default in most cube writers clusters I've seen it's unauthenticated and there is no option to keep make it authentic ated so you have if it's there and it's

running it is available and if I'm an attacker I'm kind of look at it this going you know it's not it's not you don't can't come on it but you do get cool stuff like this is a list of all the pods are running on this host and there's all version there's the darker version that's running there's the kernel version that's running there's the operating system version that's running some directory information that's quite handy and lastly some nice brush and information as well so you get some nice information disclosure out of that and that's pretty much if you find that poor open it's going to be available it's unencrypted it doesn't have SSL and it's never a center gated

as far as I can see so that's a nice good starting point at the end of the world but you know kind of handy way to next don't you that but let's top with a cube lip because the cube was kinda more interesting um there's two different poorest before I mentioned ten to five-five is read-only right read-only again the read-only port it's enabled for so that services can hit it for a management and diagnostics it's pretty much enable in every cluster I've seen it's unauthenticated on a per cluster I've seen and it's not encrypted so yeah but you and what can you get out of that you can kind of get some more interesting stuff where are we

yeah so everything it talks out things are very nice can like JSON format so you can basically hit most of these api's with a browser or with curl which is really cool if you're testing them because it makes it super easy or you can if you've got Firefox signature that's nice and this will basically this is what you get this is our authenticated stuff this is a list of all the pods running but there is one kind of thing I wanted to mention specifically which is if we search for API server this is the manifest of the API server and in the interesting part is this if we kubernetes configuration option is passed as a command line

parameter to the to the component that's just the way it works so if you get that dumped out you can get every single piece of information about how its configured which is kind of nice and handy so it tells you things like what ports things are running on what files so there's a service code key file so that for example is our private key for a net for communications now you know the file name so if you get some other vulnerability let you read files you know where to go to get it so again this isn't catastrophic you're not going to get like you know end of the game but this but it does tell you lots of

information and authorization board running in I know what ports you're binding to I know what services you're running again pretty much on every cluster I've seen if you can get to that war it's out of that community I bet some encrypted it's available so that's kind of useful okay right however more interestingly cubelet read/write this is more interesting first off the read/write port it's SSL which is nice although it uses internal it's the spigots in this case you can dump the log files at anything if you get the read/write poor now before like two versions ago which is only four months ago this was unattended by default I'm I did for another version before that

there was no option to authenticate it it wasn't possible so I thought before about start this year you couldn't have this authenticated for starters you can do this which is kind of cool because you can dump out all the the log files from the server that's quite nice but again it's not catastrophic however yeah I'm gonna go and copy this because I just never remember this come out I've stopped my head like I said before you can hit these things with curl so why should be able to do is see okay so I'm doing here is I'm posting to an API and I'm posting to the Cuba API what's interesting so I'm hitting and what you can do with it is you can run

commands right and you can run her commands on any container running on the node using the cubelet api in this case what I'm going to do is I'm going to cau you see the end commands here at the end I'm gonna cat out slash ETCs left kubernetes slash admin comm that's got the key for a third to get into the cluster so if I can get that file I can use that then to then keep the cluster game over and cluster happen and hopefully yeah that works so that's me if you get that port and you can run that command you can take it all the cluster because that's the token that ledge ticket all the cluster until

couple of Russians ago that was available but fall which is not good so if you're using all the Russians upgrade so yeah that's the first one that's like really really bad and that's the cupola the cupola basically if you get to on a node lets you run commands it's not going to be good news and you run commands inside containers so let's talk about what just containers let's talk about much is containers so we can have a condition a container that's malicious and I mention before it is a bit dispute about how to you I personally think this is a bad one because I've seen too many cases where I'm a web app test or a lot

of the time you get some form of command execution via web app and at that point you're inside the environment you can execute commands to me that's not a scenario where I should be able to go and take over a whole cluster I shouldn't be easy for me it should least be make it difficult we've got increased attack surface at this point and as a couple different ways that relevant once we're running on a container we've got container file system access so you've got access to a single file system of the continual running in which it's bad for the app but could also be back for the cluster we've got this internal network position so before I mentioned the fact that this

is a nice flat internal network that's good for as an attack as once we're in there we're in there which is great and this Kerala tax containers all run on an old obviously one of the key things about docker and everything else is you're running as the shared Linux kernel so if this is zero day in that kernel or you're running an old colonel once I've got the opportunity to run code on it I might be able to get in that way but let's talk about service account tokens because they're really wacky and by default kubernetes up to can stop it now but basically anything for 1.6 loaded by default definitely when you run up a

container in Kibera native land it gives it a token and it gives the container a token so they can communicate back to the API server so maybe it wants to talk to the API server to do something maybe it was like get some stats or talk to the API server to tell it's deployed itself or whatever else the weird thing is that by default until recently those tokens were cluster admin so if you get inside the container you've got cost robin token which is nice if you're an attacker so Boulder here is we're going to we're going to deploy container simulate our bat this is our compromised container right I'm gonna call the bad container

because you know why not so we've got our bad container and now I should be able to do cube CTL exactly well passes inside the container so now I'm on I'm on the container network I'm back at that's me I know running as my container and I'd like to say there was some like long complex elite process that I needed to carry out to do this but I hate to say that works if you've got the cube CTL binary inside your container there is a token in a default location that it knows to pick up which you will get and kubernetes handily sets environment variables which tell it where to find the api server so it just picks up the

environment variable says where is my API server oh right it's there then it goes and says can a my token yet that's there and then it runs and then your cost Ravin not very very easy to pull off if I'll talk about how to give it but basically you need to not be mounting service tokens inside containers at all under any circumstances if you do you have to be using authorization to lock it down otherwise that happens which is bad Koster so yeah service Kent tokens are super dangerous actually kind of fun someone only there's a b-side to San Francisco right up I don't know I see they run their seat they run their CTF is here in kubernetes as a cluster

and of course yeah in a CTF the job is to get flags off boxes right so they Admiral readies containers and one bright spot the first one got flag went hang on this is a kubernetes cluster got the service token got cluster admin won the CTF got all the other flags bomb in an incident because they didn't know about the fact the service to open star cluster happens that was a surprised um API server attacks so yeah if the if the so in this case and I have seen this from inside the cluster as a default option and a major cloud provider you get twenty six eight ten that's not the all core so it wasn't here forever yeah

so that you will see sometimes your CDN skip or open if you see open it's super simple you just do cube CTL - s-http

so you can just point at the insecure API server so basically if you have an app for 8080 of them you can use curl and you can do all this stuff with curl because it is an HTTP API but frankly if I'm attacking it I'm gonna get cube CTL down just as the binary it's a static go by notice it's really easy to get hold off no dependencies and just run it because it's much easier getting stuff back and the format's kind of funky it's starting anything complicated squares cubes details super easy so yeah if you see the insecure port game over again is they always cost forever and etcd last ones etcd etcd is interesting if it's if

it's open you can dump bio as well so let me get the command for max I'm not gonna remember it so basically there's a command called ECC control which you can use to dump or to interact with etcd eita bases it used to be that it was a nice HTTP API in all the Russians and indeed if you get all the Russian it is in HTTP and you hit with curl and you can just do recursive guess and scale stuff that way they've moved to a binary format now which is a bit less friendly but you can still do something like that and it is not authentic ated that saves a file called test dot DB and then you

can use a thing called bold browser it's this weird bolt format which I've not seen before I thought and you can just do that and then that's yeah that's the entire configuration of the cluster just sitting there so it's in this kind of slightly weird you can see the questions in it's kind of weird binary format so if you want to really get older but you'll need to deserialize G RPC which is what it is but frankly if you're just dumping stuff you've got way enough information to get there's tokens of stuff in there if you look around long enough so that's a really good one if he hit poor and the really the interesting

thing about a TC d is if it's authenticated it only has one option to be authenticated and that's client certificates so the way kubernetes works with etcd it's all or nothing and its authentication based on client certificate but he trusts basically i has a clustered CCA so as a trusted certificate authority and it says any certificate issued by my trust by the state court authority i trust is cool and can download the file so if you use certificates defecate or tea for a number of different things including each CD and all who can get client certificate can hit it I'd dump the configuration out so you happy we be careful with how you can figure it the certificates for that but

it's cool to attacking me where's Colonel um basically this is just standard Linux kernel stuff if you get onto a container they're running old versions the kernel you probably call that see advise or something like that you can look things like dirty care or some of the other Linux exploits that might let you break out but that depends us environment specific so malicious users are obviously much as users are harder to stall right it's gonna be harder to stall people but you might have circumstances where you've got upset people you don't necessarily want to get the ball cluster happen right you might want to say oh you can do this you do that by default in older versions

definitely that which is not starter you know your cluster app and your cluster app and that's just the way it is um but I'll talk a little bit about it because cube CTL you've seen me alright so I've used cube detail for everything here and I did see one talk where they were saying the develop that the people rolling out we're saying we don't want to give the developers SSH access to the cluster we'll just give them cubes ETL promise you've seen me exactly into container cubes ETL is SSH it's everything you can you can turn it into a proxy you can do a substation is a cool program but if you give someone cubes ETL access and you don't lock down

their rights they are basically going to be clustered but very quickly because I'll show you a little example there's a concept in container land of privileged containers and basically a privileged container says turn off all the isolation the doctor provides just give me route like I'm on the node so if you wrote a privileged container then that's bad kubernetes has an option called a low privileged which there's an API server level I can just say no don't do it don't allow privileges on in every single deployment I've seen and the reason it's on a certain system components need it so basically it will be enabled and so here I've got a wall container which it's

privileged and what you can do is with kubernetes containers is you can basically say I want to mount in stuff from the node so by default so I can just say mount in a host path of the rip into slash node and I can do something like this so exit back at my pad container and then we'll do eight once I have recalled and then we do CTO is just okay and what I've got here essentially is in here that's the that's the root filesystem the underline node and if we look at who I am Andrew so it's game over for that node out of its game room for the node unless you've done a lot configuration on your cluster

it's pull the game over for the whole cluster because the cubelet has got certificates in it which allow it to top the API server and allow it to do things to water for the API server so that's probably game over and I can add users or else I watch just by mounting in the filesystem so basically if you let someone run containers in an unconstrained fashioning use basically say I'm gonna let you do what you want the if you want to be they'll be route really really quickly on your cluster and there's not a lot you can do to stop them access to nodes are super bad basically so yeah in general that stuff not good um so how

are we gonna fix all this so I've kind of basically told you about how to break all these things and how to totally describe clusters it all comes down to what options you choose and what versions you run it's getting a lot better where I started looking at this like nine months ago a lot of these options didn't exist and basically there was no way to stop a lot of these attacks apart from lots of firewall ax I'm trying to stop people getting to the ports because if they could get to the ports it was going to be super bad knots about not so much the case now um restarting turn off the unsecured port

the clue is in the name do not enable the insecure port it's a really bad idea if you absolutely have to find it localhost on the API server that at least gives you some degree of it'll be harder for someone to get to but generally I would say get rid of the insecure port it's not good idea API server authentication this is kind of interesting right so obviously turn off anonymous authentication that's again kind of an obvious one authentication to kubernetes is kind of weird it used to be you used basic HTTP or or token off which were our static tokens which lived in a file on the API server in the clear this always an ideal if you wanted to

add or remove a user you had to go in SSH onto the API server modify the file and reboot the service this is not a scalable of abdication mechanism so these days it uses client certificate authentication you notice when I was running all those cubes ETL commands that wasn't I disabled authentication on the cluster it's just that it knows where to get the compact file to do my client certificate to pass it to thee the API server um the one problem with certificate authentication that I think should probably get mentioned is within kubernetes itself there is no certificate revocation as a concept so if you lose someone's certificate you either have to recreate the entire

certificate authority or live with the risk until it expires so either use short-lived certificates or have some other means of doing stuff accreditation you can use external authentication services and that's probably the most flexible option right now so the things that you have a web hook and you can tell the API server go and talk this way to do that occasionally shion's all the other ones are a bit kind of I wouldn't want to scale them they're okay for small numbers of users but if I had like 100 users I wouldn't be using Cline's certificates like that with no revocation because you're just gonna end up with some of this as a laptop and it's like oh smack what do we do now

that's not a good situation to be in author authentication is cannot or authorization so it used to be once you're in there was no authorization it's cluster I feel like one point fire earlier it's almost inevitably there's no authorization now there's our back row based authentication it's worth definitely that's the way to go so you want to lock down what users can do you want to start applying role based authentication to users the other thing I'll say is it can get a wee bit complicated just to give you a demonstration um you know cluster rules this is a default give me a DM install that's the list of roles and that's not all there's more roles as there are

ordinary rules as well yeah there's two rules industrial bindings and it's like what is that 40 these things so it gets a wee bit complicated so I will say that you know if you're starting with a lot users on a cluster thinking through how you want to do your authorization and getting that right early on it's probably really important because you can end up with a really complex situation and the funny thing about kubernetes is it doesn't have a list of what users it knows about so if you said to a cluster I've been off who is cumulative users don't know just to into the certificates are presented this difficult it it has the name of the user in the certificate

fields that's how it knows who you are I have what role sure you've got they don't actually know who all the users are the cost to can tell you that you have to be your CA Authenticator or whoever else manages your authentication service who's gonna know that so you can end up the BMS I think they're lying so that's yeah but definitely use our back don't use anything else use our back it's the way forward that's definite where they're heading patrol access to cubelet turn off anonymous the educational to Kuebler absolutely set the authorization mode not to always allow which is false settings and older versions turn off the read-only port and turn off see advisor

if you absolutely need these things bind them to local host far all them off but you've seen you can get information that attackers gonna be really happy about finding if you see leave these things open so definitely turn them off control access CCD so I'm basically before the certificate authorities need CCD a bit funny so you have to be a bit careful there but but absolutely trying to control access to that as well other things to think about so when I was running out that hard there and I said let it be privileged until recently that was you've never what choice to lock that down but there's never a thing called pod security policy applied pod

security policies to all your pods basically you can say no privileged no mounting stuff off the host no doing things are dangerous no adding yourself extra rights no binding to the host network no doing any of these things so you need to find a pod security policy for any pod you deploy out the cluster and get that stuff right before you start deploying things security contexts so containers both docker and everything else they kind of have an idea of what they're going to do is default they say here are your default security options you can harden that stuff down you can make it you can give the thing less right you can take away capabilities you

can add a farmer or say SELinux policies if you're rolling out across private well worth looking at that I'm not gonna pretend it's gonna be super easy to do but if you have a high-value cluster you start using stuff production it's worth looking at and the last one which is putting you in leak em out into production in the lease version three weeks ago is network policy so Network policy stops the problem of everything can see everything else you can apply essentially firewall ACL to containers so you can say I'm going to find a network policy this gives I can only talk to these containers on these ports it's not super flexible yet but it's absolutely work better than

having nothing and having this big flat internal network there across so resources there's a couple of things to look at there's the CIS gate absolutely worth looking at this is something I've started writing oh yeah this is something I've started writing so can I show you briefly oh yeah so I started doing a little analyzer tool which basically gets the CIS guy who tries to actually like turn it into a report this is a demo so it will go through and it'll check essentially pass or fail on each one of the checks which is just the wife found is oh and it also dumped out all the evidence so it says why do I think that's the case and it

I've started picking some vulnerability checks so things like unauthenticated cubelet we talked about look at that internal access API poor access the other thing I'll say about this in writing this and dependent liked rating Ruby because it's the best language if everyone likes rating Ruby a literary more that's why I phoned this complex is I memorize makes in 66 different deployment options it turns out each deployment option makes it harder to check so some of these things will deploy on with everything this containers and you can query the container and get the information you need some of them will deploy mayor node some of the cluster options like google container don't let you see some of the

stuff they manage at a cloud level so it's a bit trickier to write but that can work for certain deployment options and it's a nice easy way of saying give me a quick check to see what's going wrong I knew eventually I would end up though there we go conclusion right so why was he in conclusion to all this um the security model of kubernetes is kind of like where docker was a year ago docker had this period where it rapidly added new security capabilities and got a lot better than it was and a lot of the very early stuff about Dockers horribly insecure where the way as the added ease kubernetes to me is going

through that same process of adding capabilities and adding things that at least you can configure it securely why I would say however is that not all the different people deploying kubernetes decided to deploy all those things so a good example of this is our back so our back as I've mentioned this kind of complex I know that several of the big communities distributions aren't putting our back end yet right now because it's too complicated and they don't want to do it there's a lot work and they've got other things to be doing and there's an open source project you know they don't we're out of time so you have to be careful when you choose how you're going to

deploy kubernetes who which one you choose because some people don't you know think that's a priority for them it's getting better over it definitely is getting better or adding more things to it and always think about your threat model attack surface when you're deploying this stuff if internal containers are concerned for you make sure you've thought about that and you've applied it if they're not then fine yeah questions there are are people doing add-on products they don't do there's a lot of what I'm fighting the more is a very early days but you're starting to see a couple of cluster focuses security startups who are looking at things like yeah IDs inside the network it's a bit of a problem

though because one of the things I realized is that traditional IDs probably won't see anything on the overlay network because the overlay network can be encrypted and probably should be encrypted so yeah you're blind lower-level network censors I haven't seen anyone like there's nothing cool that's like this is definitely the way to go the first part has been network policy coming along and you can no actually apply it apply ACLs which is a huge step forward in terms of like locking it down a bit but no I'm getting like specific in terms of products yeah

yeah so if I was picking one on security and just security I would probably pick open chef rat how object because a lot of the stuff going into kubernetes from a security standpoint has come from Red Hat and they've I said you have their own implementation and they're pushing it upstream into kubernetes however obviously that if you want to go past the basic options that's a pay-for our platform so they have an open-source version but it's can like that's obviously not interfere they've gotta make money at this so their bin thing is the deploy option and they're hosted options and it's a paid for a product but definitely they have thought these things through and applied

lots of things like they had network policy before humanities-based ed and various other ones QB ADM which is one of the other like mainline options it's quite good and it's getting better and it's applying more it seems to be applying like I feel that like that's the base queue ATMs like where things should be so they're not bad the other one if you're going for a hosted option Google compute engine or cloud computing you they said Google's container one is probably the best just because it's Google's Project honestly and therefore they have a lot of people who are helping to write it so sometimes they tend to be ahead but it's a slightly different game we should quite tied into

their cloud at that point so it depends on whether you you're your heavy GCP user or not I think I don't think they I don't think they're prescriptive about it I think they kind of let you basically you know you you can you can pop you you were trying to fail past and so how you gonna do that it's kind of an implementation issue they're not if so say it's kind of funny they've obviously decided where they want their boundaries of responsibility to be and for me having a database of users and managing that would be part of what they do but they've said no no we're not going to do that we're going to pass it all by the

certificate authorities or to these external third parties so they've kind of said no that's not us so yeah I don't think they're actually like dictating the path through so

you know I swarm I've looked at quite I think I would say about if I was picking between kubernetes and swarm swarm is better at the box in that it's a lot less complex so you know all these different services I've been talking about it's been attack points then you can get your Stacey T stuck into and break swarm doesn't have any of that it's got one poor and it's pathetic is by default and you're not getting it in a hurry the only thing I'd say about swarm which puts me off a little bit at the moment is they're missing security profiles she was talking about you can apply odd security policies looked and what can be done

swarm haven't done that yet if you had it in docker run but then when they went to swarm is still in development so you can't in the open source project anyway you can't actually say I'm gonna lock this container down by tape removing so there's options we submit the shame but in terms of simplicity its way simpler and less complex and if you don't need the scale of kubernetes I would say swarms are much you know if it works for you it's a better option just from scripts that point anymore not the other example

[Applause]