Securing Kubernetes

all right okay all right welcome back to track 2 of security b-side San Francisco thank you for coming upstairs here we have Jesse end-all presenting securing kubernetes kubernetes something like that by the way we have a new prize if you ask an excellent question Jesse will give you one of these boards for the electronic badge so you can build your own so think about your question in advance for that and with that here's Jesse hi everyone as you may have guessed we're talking about securing kubernetes today my name is Jesse and all I'm the CSO and VPO at sweet Smith we're a cloud by a step product that does max fleet management so your management of apps settings etc for your

Mac Suites previously I was at Dropbox working with a few of the folks that are here today as an infrastructure security engineer all right and I realized I just started early there okay so let's jump in so what does this talk not about I wanted to start with this because there's a lot of things that I could talk about that I won't talk about because there's just not enough time so let's go through that real quick first of all setting up kubernetes itself won't be talking about that that's a whole different talk in fact it's a whole book and this is really great it's in pre-release right now on Safari Books I highly recommend it

by Kelsey Hightower second off setting up vault itself that's the secret management tool from Hoshi Corp really great a lot of tricky things here if you're setting it up not just for secrets management but also if you want to use it as your PKI as we do there's some tricky stuff there like how do you make sure you design your CI in a safe safe manner you've got the chicken on the egg problem of how do you bootstrap you know how do you get shirts from a PKI to protect your PKI when it's not up and running yet that's an interesting ants and then the third thing is getting cut off here yeah this is a really

tricky one bootstrapping involve tokens safely into your VMs and containers very tricky as well all right secret management won't be talking about that either the thing I will say is don't use to burn of these secrets they're not secrets they're just basics before not great definitely check out Kelsey hi Tara's proof of concept on using vaults within it containers as a cool way of doing that without any modification of kubernetes itself and then there's some really great discussion up a huge many many response long thread on a github issue with discussion about how to improve kubernetes secrets in the future container hardening if you're interested in this definitely check out the end to see papers by Aaron

Greta Fiore and Jesse Hertz really great stuff Aaron's paper has been called the Warren peace of container security I think it's around 200 pages long it goes through like the whole history of containerization and the ends and hypervisors it's crazy Jeff's Roselle's works on default check up set comp profile for docker was really cool eliminates I think around 50s is called out of the box unfortunately kubernetes disables that for you hopefully so if you want to turn that back on check out that github issue it's really easy to turn that back on using something called security context in kubernetes also yeah I just mentioned this but security context let you do more things than just specify a comp

profile you can also specify UID j ID GID capabilities selinux roles etc so pretty cool stuff check that off a learning detection won't be talking about that either there's a cool project from core OS called Claire this does scanning of your container images for vulnerabilities in in packages it's pretty cool and then go audit not really directly related to container security but if you do will need to do if you want Odyssey on all of your hosts where you're running your containers really really great stuff alright what this talk is about is securing kubernetes itself what is your benattia some of you may be asking yourselves this question you may have heard kubernetes it's really cool

it does container orchestration what does that mean well this is the official description from the homepage kubernetes is an open source platform for automating deployment scaling and operations of application containers cost reports providing container centric infrastructure ok what does that actually mean basically it's a tool that lets you deploy all your micro services that make up your application without going totally crazy so there's a bunch of glue that you need to actually be able to do this and kubernetes is the glue it figures out which machines container should be placed on how many should exist handle things like scheduling load balancing etc also does data volume volume sharing so if you need to share data between containers

helps you do that if you need to share data from a host to a container vice versa helps you do that as well also figures out if it's actually safe to send data to a container is it actually healthy is it up and running can it take traffic yet handles all of those things for you as well and finally it really makes deployments a lot easier so if you have you know 30 micro services and they're all version differently and basically you want to do a release right maybe you do that every day several times a day every few minutes once a month who know it doesn't matter but every time you do or at least you want

to know that everything up and running if it's not healthy kubernetes can actually help you stop that deployment automatically enroll back to a previous deployment of your containers using a feature called deployment so it also helps a lot with just the deployment process in general so these are those some keywords you're going to have to know throughout this talk I may have already said a few of them a node is just kubernetes term for a host probably a VM if you're in the cloud a pod is a group of one or more containers and a service is an abstraction that lets you say here's a service let's say it's your nginx at your edge what happens if those

containers die and come back you still want to be able to route traffic to those new ones new pawn new pause that comes up right so this is just an abstraction that lets you say always let me address this thing at this address all right there's two types of proven ideas so there's a master and the worker outside of that is that CD this is where all of the clusters data stored it's a distributed key-value store so you just got a CD running a buck on a bunch of em and we'll get into where that fits into the picture in a second master components there are three main components to the master the controller manager the API

server and the scheduler the API server is the real big boss that's where most of the magic happens totally stateless it's just a REST API that everything else talks to the controller manager enforces the state of the cluster so you describe the state you want things to be in the cluster manager talks to the API server make that happen and make sure it stays consistent for you you've got the scheduler this is purely responsible for scheduling those pods those sets of containers or call your hosts that's it for the master and you've got the worker a worker has three components as well arguably only two up to you whether you consider your container engine as part

of kubernetes or not obviously it's pretty critical that your docker or rocket or whatever you're using to run your containers technically not part of kubernetes itself but it does run on the worker host couplet that's the main the big boss for the actual running of the containers you know the pause themselves that's the main agent that's talking back to the API on the master he's got the proxy this is responsible for directing the traffic for different services to the right location is also talking back to the API server so what does this look like when you put it all together it looks like this stunningly this does not exist online anywhere that I could find not with the actual service

names there's abstractions on all the other charts that I could find so this is this is how it actually works and then of course there's actually a connection here as well between the container engine and the COO blip as you can see when you when you start thinking about security here you see as obvious route of trust is SED for the cluster that's where all the state is stored so if something can talk directly directly to a CD that's bad news same thing where if everything could just do any arbitrary API command that would also be bad all right I always find it helpful before actually trying to secure something with TLS to actually look at where everything is

listening this was also really hard to find a complete list so I created one start with SPD is listening on two ports one for client operations and one for server operations so client meaning kubernetes api itself is going to be the client in this situation talking to Ed CD encore 2379 at CD itself when it's replicating all that data from one host to another is using a steering port which is 2380 a quick note on the master before we go on to the next slide there's a bunch of ports that only listen on loopback and that's for the other master components the scheduler and the controller to talk to the API but since those aren't going

over the network kubernetes at least in its current version just makes those what's on our loopback there's no security at both if you have someone read on that box you have bigger problems and then you've got the public ports and those are the ones that I'll be talking about here so here's where it's publicly listening accessible on the network the meaning API port I'm putting what you should this out there for the API server I believe the default is actually 443 and I'll explain later on while you probably want to change that to 6 4 4 3 or something else the scheduler got a health check port and the controller has a health check as

well alright the work reports the couplet which is again the main kind of master process if you will on your workers is also has an API and that's on 102 50 there and then you've got some you've got a read-only port which you can disable if you don't need it the adviser port which you should probably disable if you don't need it it's just statistics and information but it's kind of unnecessary and then the coop proxy you won't see anything in that port range unless you've created a kubernetes service and that's what I talked about earlier when you need a consistent way of addressing your pods all right so that's an overview of kind of all the different components

where things are listening let's go ahead and move on into the authentication authorization here so there's sort of three stages to to the flow here the ones you probably are going to care most about and that I'll focus the most on today are the authentication and authorization stages but the admission controller stuff is it's pretty interesting as well and you'd want to watch out for that and I'll mention why at the end all right authentication so there's two main types of accounts is user accounts and service accounts there's a pretty big difference between them the biggest is that service accounts are stored inside kubernetes itself in in a CD and part of the clubs or state you can create those accounts

from the command line using the coop control command line tool you can not do that for user accounts those have to be specified in yamo that's probably the biggest difference for user accounts you've got a few different options for authentication this is not an exhaustive list of everything that's port but these are probably the ones that are going to be used most frequently the one I didn't include that I probably should have they do support open IDs so if you wanted to you can have your users authenticate with you know their Google their G suite credential q kubernetes if you wanted to the one we're going to be interested in is the client's Earth's surface account

can't use anything but JSON web tokens directly the way they secure this if you do have a CI a and you want to secure everything with TLS is you actually configure kubernetes to sign these tokens with a with a private key so that once they actually reach the API server can validate that it was signed by the same CI alright authorization so there's two main options here there's other options that you probably don't want them it's like denial or like a all the 1p we actually want to use are probably just one of these two so a BEC is the current implementation the current default and they're moving to our back I think in the next release one

point six is coming out I think in March I think March and the admission controller I promised out mention this briefly that's what actually happens after authentication and authorization have already completed there's a check that gets done the is there any special magic that needs to happen here before we actually write this to SPD and this is where things can potentially get dangerous I'll mention why in a bit this is what the config looks like for a vac the current authorization implementation default in kubernetes this is an example of the role based access control that they're moving to quick check-in of what we've covered so far we've gone over the kubernetes components where they're listening we've gone over that chain of

trust sed to the API server to the couplet as well as how authentication and authorization work in kubernetes so this is all very academic and interesting but what's the actual threat model that we're trying to protect against here for me this is the most interesting might be different for you but for me okay they've escaped the container they've got a show on a worker VM what can they do from that worker obviously don't want them to be able to connect directly to FTD and modify any of your cluster state you don't want them to be able to man in the middle traffic to and from any of those other kubernetes components and so those are

some of the things that we can protect against using TLS so Q Luxor can be used to kind of solve all these problems for us you can use them for authentication they can assist with authorization by if you embed at the group name if you're using our back into the search then during the authorization stage it'll actually look into that introspect into the serve and pull out the group name and then apply those correct permissions so that's pretty cool and of course transport encryption so how do we actually configure kubernetes for TLS I'm going to walk through this pretty quick I just included these mainly so that they'll be in the slides later on you can you can look through them so

obviously you need to specify a CDS config probably the most important for the after me after mentioned reasons keep in mind that you're going to want to specify both the key pair for clients at CDN client mode and in server mode right this is the main API server for kubernetes pretty pretty clear-cut here these are not full config files I just pulled out the TLS related stuff so you've got your couplet user there you've got your scheduler in an ideal world these obviously have separate sorts and keys right I want to go through this step by step because there's so many different TLS config options in the file it's kind of hard to see what's going on so the first part is

when the API server is acting as a server we need to accept incoming connections from the couplet that's running on those workers so that's that config there we've got it the other direction so the API server as a client connecting to the couplet on the workers you've got acting as the API server acting as a client to FTP here at the top and then I mentioned earlier if you want to actually secure those herbs account tokens those JSON web tokens you have to sign them to make sure that the API server actually knows that they are from the same CI and finally unless you need this for some reason you probably just disabled the anonymous

authentication so this is like nothing required there is some amount of information you can get from that why expose that if you don't need it scheduler config this kind of pulls in all the same config that was specified on the earlier coop config file this is a this is that file actually the coop config and the couplet this is again the part running on the host or sorry on the workers and the proxy also pulling in that same coop configure okay great so that's all fine and dandy we know how to configure it now but where do we actually get these certificates from so we get it from the vault PKI and once you've done all the painful

work of setting up vaults and setting up this all PKI which as I mentioned is a whole separate talk then it's as easy as this you just do a simple occur l-- include your vaults tok in there and then you specify you know what you want the CM to be the common name so in this example we're graphing one for the cou blip and we're just counting this to a temp file doing a little low parsing and then removing that temp file i forgot to mention one thing that's relevant to this this bit here which is that you have several options for volumes within kubernetes so you can store this on a memory only volume so this actually

isn't written to disk it's just stored on a ram disk essentially which is nice so what have we accomplished at this point we've cured SED the api server the scheduler the controller and the coolest all of TLS which is wonderful there are some limitations in kubernetes itself right now that are a bit disappointing there are workarounds which i'll show on the next few slides but the biggest one is that we haven't secured the connections from the api server to the nodes pods and services and unfortunately this is not possible to do with TLS with fewer Nettie's currently i assume that it will be possible at some point but i haven't seen any indication of when that would be

and that's mentioned on that page if you're curious where i got that another huge limitation and this won't easier workaround but there's no support for specifying an OCSP server or ferl file and kubernetes at the moment kind of subpar solution so for the first issue that I mentioned one solution is securing all the stuff with a sage tunnel I'm not thrilled about that but it is one option and this is actually what Google container engine does to protect you from this they do a bunch of stuff to secure kubernetes way more than you would otherwise get out of the box out of the box you don't get anything it's all plain text so that is a good

reason to to use container engine if you don't want to go through all the hassle on this stuff one solution to the no revocation problem is obviously just doing short-lived search with a aggressive rotation so that's probably the best set right now and maybe that's a better way to go anyway the other option is instead of doing this at the authentication stage you can wait until authorization essentially and just you know kill that user account don't just remove that user account from your configs unfortunately I believe this does require restarting the API server which kind of sucks because now you have to schedule time to remove accounts I hope they fix done additional work that

you want to do that I didn't really cover but I'll briefly cover it here if you want a secure kubernetes deployment you probably also want to secure the communication between kubernetes and the container engine for example docker with TLS as well you would want to give every user in service account a separate certificate make sure you configure proper permissions for each of them preferably using our back like I mentioned coming up very soon I think it's in beta currently or it's an alpha now and it's moving to beta I think in 1.6 and yes the admission controller so I mentioned I would come back to this this is pretty important by default the admission controller for service accounts gives

every pod access to a what's called a default service account role and so that means that all of your pods are just using one universal account to talk to the API server obviously that's really not great you can turn that off and then you can issue individual tokens you know per per role or per like group of containers of you per pod sorry that you have which obviously would be a lot better than than one for everything additional things you should probably do is disable that see advisor listening forward unless you need it I mentioned that earlier disable that read-only uh Nautilus access if you don't need it and yeah the reason you want to do this is

if you are running kubernetes in production as we are you probably don't want people to be able to run tube control commands from outside of your production environment and obviously if it's listening on 4:43 unless you work somewhere where you don't need to allow 443 in coming to production which i've never heard of a place with that type of setup you probably will want to push the listening port for kubernetes so that people can't just run these commands from their laptops totally bypassing your SSH Bastion so six four four three fifty some choice for them also yeah that's default amazing default second profile that just Iselle created while she was a talker that's totally disabled by kubernetes

probably want to turn that back on I was tested very extensively at docker so it shouldn't interfere with anything and it's just you know you basically get it for free it's really easy to turn that back on just a tool unchanged and in your yellow file for the pod and the config to do that is on that github issue all right so here's a recap of everything we talked about we went over what kubernetes is all their components the network connections between the components our threat model sort of talked the critical connections between those components and how to secure them with TLS example configs limitations and workarounds with kubernetes in terms of security and then finally the additional

work that you should probably do after you've implemented TLS you've got that laundry list of other fun tasks to take care of if you want to secure kubernetes deployment and I think that's it I have two more slides after this with a list of a bunch of links that I use to put together this talk so they'll be in the slides as well that's it yes in disabled yeah yeah the question was he mentioned that kubernetes was being used for the CTF here and the default service account wasn't disabled and the question was why would you actually want a single universal default service account I think that the answer to that question is you don't and the reason they did

that was I think just sort of for speed during development I don't think it was a conscious like we're going to do this after having thought through all the security implications of it in fact there's a big discussion on this on a github issue and basically everyone agrees like yeah that was bad we shouldn't have done it that way they're fixing that I believe in 1.6 they're moving the service account to our back and presumably scoping down the permissions that the default account gets I I don't know if they're actually disabling that automatic issuing of like the same account for all services but at least it is going to be scoped a lot better than it is now question

yeah so the question was if you are not going to use SCD for secrets what should use because you obviously need to put secret somewhere I do want to clarify one thing there which is that specifically the thing you don't want to use is kubernetes secrets and kubernetes stores the secrets to net CD yes but the issue the issue at least for the purposes of this talk the issue isn't set at CD itself isn't storing those secrets encrypted on disk that's a whole separate issue but the issue is just the kind of scoping of of the role that's used to access it but the tokens as well as I mean it would be great if they were

stored encrypted obviously and so sorry say it to answer your actual question here what do you do today so what do you what you do today is just use something else for secret storage so we use vault as a general secret management tool as well as a PKI and that's definitely what I recommend that's the thing I mentioned earlier about a proof of concept from Kelsey Hightower using Annette containers to basically interact with walls and gets decryption vaults that's what I would look out there yeah that gets to the chicken in the egg problem of how do you secure vault itself which I'm happy to talk to anyone afterwards it's kind of a long discussion but it is possible to do it's

just a little dance you have to do yeah so the question was basically why was the default docker second profile disabled by kubernetes as far as I can tell there wasn't a great reason for that I think there was some general fear that maybe it would interfere with you know with people's containers but I think that fear was maybe unjustified because there was a lot of testing that was done to make sure that those wrote that profile wouldn't interfere with I think they tested a whole bunch of dock containers and you know got for the point where nothing was nothing that any normal Microsoft would need was being blocked so yeah there is actually another github issue now to turn it back

on but yeah I don't think there was a great reason for turning it off other than kind of fear yep pretty weaving Oh is there a plan to make kubernetes gui-based there is actually a GUI now I think they introduced it in 1.4 I don't think you can do everything from the GUI yet but that's definitely there then you get into the whole how do you know now you have to control access to that as well so I guess you could probably have both you could you by the way you can have more than one authentication type enabled I want so you could use client certs for people who are you know within production yeah you could use you can

define us but then or clients routes within fraud but then you could use you know open ID before for maybe the the web UI portion yes exactly yep all the other PKI so you could use of involved there are I would look at acne CA so that's the one that's used by let's encrypt and the core TLS operations in that or handled by a CF SSL CloudFlare SSL and that because it was being used for let's encrypt they actually ought NCC crypto services actually autos with all of that code so it's probably pretty good so that's another one if you do want to be brave you could implement you could use CF SSL kind of as a library

and sort of build your own CA on top of that just like acne did that would be a good starting point that's obviously going to require a lot more engineering hours but yeah cool looks like stuff so we have time for come find me if you have more questions thank you all right thank you very much Jesse before you leave a Fitbit for you from our sponsor thanks very much all right coming up next we have Amanda Russo in about ten minutes so Amanda is she in the house yet no all right ten minute break come on back a reminder there's a happy hour tonight 520 sponsored by Salesforce

[Music]