Introduction to Container Security in Kubernetes

talks to the intro to Danner security to regards to kubernetes most of it in cover names Oh actually before I do that let me just finish I just want to Rome Fort Kiowa sequin speakers do this but a show on hands if anybody has any experience with docker and kubernetes and all ok how about just Dockers and itself ok so I try to get through all these slides I have a blog really horrible making presentations bear with me it's a little bit about me like I'm good for short your architect like I don't really use Visio inside with my co-workers I'm really horrible and pictures and anything that looks like nice long presentations I'm really bad that you

know wore many hats in my career I've done a few different things within this world you know that's my tone dial tone I have the day for remembers bad thing is this you know God was really into stuff my first Def Con I directors Def Con 9 so back in the day like spark this is you know I read a lot of pulse one in particular that game's popularity was cute ps1 all right here so if you ever you know running kubernetes and you need to check out better context an aid station with your on the command lines or really good utility for managing yet so give it a you know give it a welcome to be able

watson and also big balance or maybe wants to talked about spelling with the viewers you know it's a place to start guys this will be by Genda we're just gonna go over I'm gonna do a little brief history lesson on containers I like to do the history briefly just because that's that's all I learned like I always go back to the history go there to see working from and how it was designed why it works the way it does so I understand all the different stress abstractions more clearly I'm gonna discuss a couple of the potential attack vectors of every they face in the container world especially within kubernetes I'm going to talk about the

different parts that make up kit Bernays and go to details about the control plane worker nodes and then their specific workloads within them and we're going to auditing the logging and then just some suggestions best practices on the security of UNIX guys right there doesn't even in some of my picture it's a pretty free fishery to contain universe so they're not really do like they did like that old reputation like the cool kid thing and like you know all those of our stickers on our laptops like they have you know docker stickers and everything but they've been around for the technology have been around most of the stuff in computer science was done and a lot of stuff computer

science is pretty much done 70s images extracted for the last 30 years so you know 1982 they had the the churun system call and that's what it really started it was just isolation on the file systems they got later they later and Todd became known as jails and actually in the code now it's actually referenced as jails in the PSD code if you go home within there the seeker has that listed and then a whole bunch of years later Solaris created zones now that was the first commercial usage really of this type of isolation in the wild a little bit after that was that openvz project it did really gain much traction because it was rather complex but a little tough

to use and then in 2006 and sevens when everything changed because process containers were created beautiful at the time I was looking for ways to condense and really maximize the utilization and in their data centers and they saw these control or these can process containers being created so they went higher all the current developers that were designing these and created their own internal cluster scheduler all the work that I'll get into later and that's that's basically what kubernetes is based off of but a lot of this research a lot of the technologies that are in place right now are based off of years experience inside of Google and how they run containers and scale and then as you

can see secrets were merged into two 6to4 control process containers were later renamed to controllers 2008 LX containers I think some people probably still use I like see but it's it's a good implementation of namespaces and see groups but it's not as fancy or as user-friendly as docker and that's what made docker so powers what was you were able to take a command-line that was like probably thirty arguments long that was necessary to create a container and to shrink it down to dr. Rafat and that just abstracted everything out and made it so easy to run containers and has the popularity the growth of experience since then is there a for how much you haven't played with LHC is there like a

is there like a docker hub for galaxy cuz that's probably one of the other big things for doctors I don't know I just like touched on LX I looked at l XE woman when I was doing the research but I didn't go into much detail but that's why I write like I think my daughter's popularity really gained from the ease of use like because if you look at the clip like the first implementation was working Python and if you reverse engineer that code up you can see like the system calls that are being made and they're they're big where I like you have like n Center than all the different secret stuff that is necessary

to create the isolation and it's just all taken away from that there's such simple commands I was gonna do a demo of that but I 39 so this is another history lesson here so board is the cluster scheduler that's internal to Google and mega was the one that came after work internally and then both of those were pretty much merged into kubernetes as you can see here so kubernetes is an open-source version of what Google we did and currently does internally in their data centers every single thing inside the Googleplex like absolutely everything no joke when I say everything runs inside of the container when you go to Gmail where you create a Google Doc board will spin up what they

call in Alec which is the same thing as a docker container and you will get your little instance and you'll be able to edit in Gmail and they don't give out a lot of detail there are many good papers that link at the bottom there is an excellent paper about war it's like Google releases white papers all the time it's like a 30 page I believe 30 or 30 something big paper by the engineers that design more and just like I even know how many technologies are out there now and as soon as that white paper gets released usually a technology comes out to rip and then it gets adopted like Cassandra the same thing happened with it

do other technologies respond that way but the difference here is kubernetes was actually built by Google engineers and a released open-source into the wild and it was the number two open source project at in popularity they had the most commence the mode it had something like 75 years of human development time within like the first two years of it being released which is impressive you know I think the only other project that could bypass that would probably be the Linux kernel I so for the uninitiated what is the container right so you don't hear this word all the time it's basically it's just a stripped-down version of the Linux operating system a lot of people call it OS virtualization

OS level virtualization but it basically just uses the hosts kernel system cause that's all because it's not a VM it's not an entire system it's just encapsulation for a single process 1 process of Tomcat as an example bronze is a single Java process rights you have Java - whatever run Tomcat and that gets wrapped around an allocated a certain amount of memory and access to the file system that's all in containers it's not really there's a there's a really clever engineer Jesse Brasil she always says like a container is not a real thing it's just a bunch of system calls wrapped around a process and that's that's basically all the container really is there's like three main

technologies that drive the whole container world secrets namespaces and the unique file systems go underneath that if anybody wants to talk about the technologies of docker and everything I can later and stuff but they're not these extremely crazy complex entities it's just wrapping resource constraints around single processes you see your difference between a virtual machine in a container with a virtual machine you have your infrastructure then you have your host OS like whatever it may be and in your hypervisor ESXi HP kvn any of those hypervisors and they can build your OS on top of it traditionally and then you launch all your you know your libraries your by there is everything you need for your applications to be

able to serve traffic and to be available to customers right and the potato roll it's a little bit different than that that's not necessary for any of that compute you have your infrastructure your OS and then this picture is a little dated that has the dock branching you can replace that pretty much with any container API because docker is not the only one anymore in fact docker is actually more of an ecosystem nowadays than it is at the container runtime the container a lot of time itself is now also extracted out of doctor or doctors like a full tool suite and is the blue lighter that says daughter engine just be the container API basically I like this image it just

has like the old way versus a new way you know like and the old way to deploy virtual machines all the time everything we need you know we're sitting there worried about y'all updates and afghans and everything trying to keep our applications up vulnerabilities you have to like deploy however many updates to however many nodes and may manage those operating systems and everything that they're doing and then with a new way with containers you don't have to do that at all like you maintain your fleet of machines and that's all and your applications sit on top of that self-contained moveable destroyable and reproducible within seconds as opposed to through the entire VMs or entire operating systems all the time you

always hear people say containers are super fast to boot up that's that's it's weird to say that because they don't approve like it's actually just like it's a process execution so that's why they boot fast because it's just it's like typing top at any enter that's like essentially all that it does when you rocket and that's why 13 sighs all right so before I get into all of the intricacies of kubernetes I'm going to go over a little bit of the couple of the attack vectors some of the popular attacks and vulnerabilities that have existed in the kubernetes these are darker specific these are going to be compared any specific attacks and these are the main ones here some of them are

easy to configure some of them are pretty complex you know but and many was are going to be container compromised obviously because the container runs on the host if you compromised the container and you have access within it you can sometimes escalate and get post access another one is unauthorized access between pods described of pods are later just but real quick so it might make a little more sense apology just a collection of containers that's all there's just one or more containers to another layer of abstraction kubernetes adds it's just saying everything is everything in this world is just abstractions data exfiltration of course you miss configuration I was going to put like another studio meme on you but I'm about

doing it miss configuration is always like you know the one on our side cuz you could give somebody access to a system with sudo thinking you're preventing certain attacks from happening or securing their system but actually makes it worse it's not configured correctly same thing can happen here if the systems are configured correctly you can actually open up bigger goals and then direct access to the house so like it's always a good idea to have you're like asking hosts or boxes you know in this world it's pretty important because you don't want you have access out you know keys of the kingdom to the hook to the cluster itself you know that old like you know you know if you should have to

SSH into every machine they don't look at it it really applies in this world cos if you have access to a worker node in the end to the next cluster you have access to all of the containers running so if you have room on that machine you'll be able to see absolutely everything that's going on on all those containers okay these were the two most damaging ones I think in recent times these two CDs the first one basically allowed rewrite behavior the files that were specified within the pod even the host file system so the attacker could just be inside of the container and have complete access to the host and do anything they wrong with it and they

since fix that by just changing the wind at the cell path works inside kubernetes there's a link at the bottom there that describes this attack of detail I should reproduce it because there's no exploits word it's like it's just a series of commands so there's no you know there's nothing necessary to exploit these other than like doing some symlinks so they're pretty damaging but it's configured correctly they're they're easy to avoid second one was ridiculous because you can just simulate arbitrary files have been like a container and then delete things on the host you know like simulating as a shadow into your temp directory and then you can just take a look at it best has since been fixed that link is

at the bottom all these are very reproducible if you run like a mini queue breaking locally you can try their attacks out it's a penny or out of all virtues like 180 vulovic arrays and you can like perform both of these attacks for ten minutes probably anybody remember this attack at all sorry about my Tesla they got hacked really bad because dr. Riggs dashboard was left in security no password access whatsoever and back before to raise one body the dashboard had super user access to the cluster and because of that they had their AWS access tokens so the attackers started using Tesla's infrastructure and AWS - it's pretty bad I think I have a link at

the end of this for this but just prove like the Tesla Tesla hack it was ridiculous like how easy it was you didn't really even have the hack that then you just went to that URL there was and you can see on the on the side where the workloads are over here you have access to yeah so I'll get into that little bit later on how to keep this secure but this has changed drastically the dashboard no longer has unrestricted access into the cluster like it used to thank you all right dividends I like these cheesy means but I guess sake it's like kubernetes is pretty complex you know there's a lot of moving parts within kubernetes and it can

sometimes be daunting but if you look at it later by later I guess you would say it's not as complex as it as as it might appear it's basically a container Orchestrator it takes docker applied to many different hosts and you can cluster it it's completely API driven which is very very nice you have large-scale distributed computing with an API as completely unsuspecting scale it's a very robust system there's unbelievable amounts of documentation out there on how to use kubernetes how to deploy kubernetes so it's not as you know it's not as bad as what what some people I think it is you know if you really want to learn the details of it there's a kubernetes the

hard way which is a pretty good exercise to go through I did that before to learn the different components you basically get to operate containers instead if anybody has run dogger even doctor locally it starts you know a few docker images on your hosts not bad right like you can have a couple quarts of them you can do some port forwarding or you can run it with some load balancers to get to those back-end services but once you started scaling that out you have 10 15 20 30 the containers it gets a little bit hard to manage due to the ports kubernetes takes care every single unit of compute within kubernetes gets assigned individual IP addresses so you can have

all of your applications running on the same set of standard reports as they do within virtual machines or bear that or you can have Portland have your application run on port 80 instead of you know 80 80 81 82 because you have four to five thousands of times yeah that's why that's the architecture of kubernetes so I'm going over all these components in how and how each of those can be secured and what primitives are offered by could raise the key BG's components secure but the main the main ones like so over on the my right is the UI that I just mentioned it was completely left over with my Tesla and how the UI interfaces with the API

through the API server directly talks to the masters and then a master is just the brain of kubernetes that unit face word for everything and he talks to all of our minutes I'm going to describe the masters the worker knows things to do with those to keep them secure and to lock them down correctly and then and also the workloads that are deployed out on there so he readies offers a ton of primitives that are built into the system for security there is I don't mention anything in this talk of third-party tools everything that's in this talk is all native to the environment and all native to to this world I like you mentioned like some applications to

automate the testing but other than that there's no products or anything mentioned in here because they offer so much in their API to keep this things to keep it nice and secure yeah so the great cure that is the control plan as main part of the master server they call the master/slave relationship terminology they use this is usually just called the master but it's the control plane and it runs these components there's usually four components but five you run in the cloud that you know the cloud controller manager which is a nice that's a nice service because if you're writing a document gke or any other cloud provider that Cloud Controller manager has hooks directly into those api so if you spit

up a service and you need a load balancer that Cloud Controller manager will handle that for you and actually talk to the guys and deploy the balancers or any of the other service type features for you automated which is very nice NCD is the key value store that is used for all of the data it's a consensus cluster system so it it holds absolutely everything necessary to keep this cluster on I talked about a little bit later on how to keep that secure because there's a lot of sensitive information it's in there that's absolutely everything that the cluster needs is within a CD yeah that's really positive a bad addiction pathology kubernetes worker notes so I'll eat your

worker notes in your environment you have these three components that run on it you have the cube with the cube proxy and then you potato runtime I said before the docker is basically like an ecosystem now it's not really just four containers right the container runtime is an abstract 'add daemon that talks to whatever container api you want to use so if you have open shift clusters you can use red hats exquisite remember they have their own cluster around the container API there's bronze seed which is the default within docker this container key which is now the default within to prevent is that you deploy kubernetes like in gke yoga was called container B and none of

these have daughter because doctor is too heavy now because there's too many components and when you run your potato cluster you don't really need docker because when you're running a container you don't need to do a daughter build you'll do not build machines so they don't have that anymore deployed in the clusters there's like a nice little thin layer that just has the fuse a few cron calls that are necessarily create your container destroy the container pull down the image update the image and delete which I think they're pretty much the only API calls that are necessary for that container time to take place any questions about that exactly a little bit confusing because doctors always the main thing

talked about and I mean you can run doctor in here but it's just not necessary because it's just such a heavy API now and it has so much tolling like there's also daughter swarm which is a cluster instead or but when you deploy doctor you get thought or smore so you have all these components running within your cluster that you don't need so that's why they came up with this nice container runtime to be able to abstract that away they've actually done that every component in this ecosystem they distracted everything out putting a pion layer on top of it too so you can decide your own network layer has it you can run whatever virtualized network

stack you want with enterprise there's a cisco makes one there's open V switch everything is pretty nice and distracted it's very very nice it because it can run anywhere to kind of further like the company behind docker also kind of change the name from like the docker and into container D but there's like a bit of a name without doctor are they changed their entire ecosystem they make off like the bill for dolls like Modi and then container D they just that's just the name of the binary that interfaces with the car it was at one point dr. D that dr. mg and dr. D and now there's container T but that's not container D I'm not sure

it's either potato peer-run seen as the default that's provided with docker it's one of the two but those are just that's just the actual API layer of the toxic economy alright so those are the main components and then the first thing to talk about with in keeping this thing secures our back right role based access control so I'm going to Cooper day's 1.6 they started adding our back in enabled by default and also with that deny by default so if you would back when kubernetes more than six first came out we got people not dead by it because they would deploy their cluster I got 160 miles let's update you know nothing from one five one six all of a sudden

nothing worked because everything was in everything was in and denied so you had to start enabling every single piece of communication excuse me but in the cluster which you wanted to happen the our bag has rules that apply to users they have it gives permissions that they could perform of in the API server and then you have the questions which are plus to watch so roles apply to specific sets of resources within the cluster and cluster rules apply globally to the cluster there's a little picture there to help clarify a little bit because sometimes like even at work from sometimes write this on the whiteboard because it gets a little confusing we're like ladies enroll body visitor rolls at our

cluster roll so I can write them down every time like truth charts on the whiteboard before we start like it's a figure like oh we need to have this rule binding and then I break everything and [Music] you can see that the you know a roll classical resource rule binding applies to the roll and behind that is the entity that were close so a pop that space this is an old picture it just hasn't worked doctor in there but you can replace that with container engine basically pods are collections of containers deep dive into a pod and you'll see that there is a single container running there called the pause container and just runs a system call

called pause so if you're ever on a container cluster and you do a dr. PS there's always this word pause everywhere that's just a nut that's a container that adds a layer of abstraction on top of containers to provide the ability to have multiple containers within a block so that's like a little bit of like Inception but basically you can have shared namespaces with in doctor but you can you can share names basis so that's all that a pod does is it has a shared namespace and attached as other containers to that shared namespace and it does that with the pause container and then has the ability to schedule and do everything as single units so a pod

you can think of as basically like your application stacks so like the pod can be located or the containers that are in the pocket you love they need to be so if it's a lamp stack you can have your Apache process - my sequel process and your PHP stuff running all of in one top and that would be three different containers but they're all extracted and ran as a pond and we can get rid this is you get a single API site or a single IP assigned to each of these instead of having three individual containers and portho me on it and all kinds of different rules this abstract set are weighing creates like nice little cohesive units that can now

be moved all over the place within the cluster that came out of that idea came out of Google because they they obviously have a lot of gears and they could not do the translation stuff like to manage like 441 one computer so they decided that this was a better model by assigning every single unit a and IP address

it's not a pod level security so there's a lot of stuff you can do with the pod level you can limit everything that the pod does so you can tell it that it can only you know get an list and it's all API verb actions so everything that the pod can do you can restrict and limit access to a pot reader role in the role binding like I was describing before this has a role binding pod reader so if you go back here so the role bottom applies to that entity which is the user or the group and then that applies to the povery or Bowl which gets assigned to the puck and then it has allowed to

do what you tell it you want it to do so you can test it you can say that the containers within this pod they're only allowed to do you know XY and Z and do not let it do anything else it's actually quite simple there's a lot like I said there's a lot these are all the available contexts that you can add to a pod I wasn't going to go through all of these because it's like you know I'm like giving a lecture but there's a lot you can even add node level contexts within your pods like selinux SATCOM app armor and capabilities you could also use those and put those primitives into your pod to be able to limit what it

does right here so if you get selinux and capabilities Apple our SATCOM sure everybody's you know vaguely familiar with that we've seen how to run you know set force to zero but they're really nice like if you apply these at the pod level they get very powerful and to really help with whatever working outside really secure so this is the this is an example of a pod definition within kubernetes so I set the security context these sides right there on line four and that gets applied to this so there's the spec that just says rod is this user 1000 has to be has to be a number cannot be a strength for the user I'm telling

it I only want to read only file system because this is just a process product I don't leave your my data it was in this guy-- so you know somebody were to privilege escalate within this container they won't be able to do anything you know I can't touch anything on the file system you can't make any modifications whatsoever and you're also denying privilege escalations so if you have like two layers right there so you're not running as root you're not allowing anything to happen on the file system and privilege escalation is not available so you can give it anything expert once you're inside of that container that's just 40 give examples of these guys all being applied to that

particular bar there's a lot more that you can add like I said with the selinux and privileges or on the capabilities rather than the set comp with all the different things this is a new topic I just want to touch on like briefly this is one that's getting pretty popular within this room it's like the idea of sandbox pods there's two projects out there right now one of them is Google's chief Iser and the other is called cata containers it basically just provides like a whole nother layer of isolation rights like I said for more abstraction so in between the kernel you know you have your container engine and then after the container engine you're going

to have your applications running a larger system calls are happening so with these sandbox pods you're actually going to move that nother layer to user space provided shinmin between other thing and then have that layer of isolation so it's actually quite nice like I ran GE buzzer locally for a while there's like really no changes it's a slimmed-down really really real really small Linux kernel that only has the system calls necessary for containers to live so I mean there's like 340 I use system calls come for you know the container doesn't need that particular cops but he does all the system calls you know he's like 12 so the g+ just slimmed down colonel and

only gives you access to those same thing with with the calculators it just adds a little shim in there it gives you the runtime for kappa on top of the container a long time and it's pretty nice that you get like really hardcore isolation so if you were to break out of any of the workloads and get access into the container they're going to access system assistant level kernel or a user level kernel they can't do anything except maybe my starter container backhoe so it's not really damaging at all yeah so the next big piece is the network policies this gets really like very granular and there's a lot I can talk about Network policy

products for an hour this is just one example right here it's basically you're going to specify what a pod can do and every single layer possible so you can tell it that it can't talk to the other containers within the pop you can tell it that it can only talk to one container in a pot over there you can tell them how to talk to anything because granular jiwon and as security is over this there's a ton of policies out there that are already written and available that he's a Google engineer has this repo with these images that is actually a gift that moves I'm horrible like probably good slides but these like dots they move you can see

like it just says that in the default namespace in the middle the web app to talk to the DB but nothing else to talk to so all the other applications are arriving in your cluster they can't even talk to it they're limiting it that's it there's a little prospect that I wrote never policy rather this is the animal that has the pond selector says why need my applications called besides Delaware it's an API server and I'm only allowed address to that pod anything that I'm able with besides Delaware so nothing else that's in the cluster unless it has that label applied to it and talk to this that's that's basically how it works those rules that are on from five

down just define exactly how the network policies work you can create these at the namespace you can create them at the pod they can be globally scoped and they give you every sort of resource close so another ton of stuff so basically like a resource quota just lets you know or lets the cluster know how much compute want to give a certain container that's basically all it does but it applies to everything so it can apply to the CPU and it's also the limits of the requests of the CPU applied to the memory you can apply it to a part of service every single piece of the cluster you can apply the source close to and again you

can apply this to a whole namespace you can apply it to all your pods if you want to get applied in one part so you can put the definition within your palm so yeah like so say an attacker gets ahold of the pod or your container somehow bricks out of it is able and to all the other containers within that pod right and you may try some DDoS start Bitcoin mind or something this having these resource pointers in place will not allow it to be able to consume any resources extra that you tell us so you can say like you know you can baseline your applications find out exactly what they need and then you can

assign that to every single one of your applications in the cluster so you can maximize your clusters footprint as much as you'd like by analyzing all this data which is all available by the way so to read eases follows that EDC methodology will always be collected it has every single metric it's all collected all the time with by the API so there's an example of our resource portal I'm just telling it I'm given a 10 gig of memory 500 milli see you which is like a half and it rates it like AWS that's like a like a half a game and then that's I could do that with the request but then I'm limited to

that same amount so we can't go any more than that as well basically all that does that's applied to the pod you see line to supply to depositing and logging there's three different styles of auditing and logging within the cluster you get out-of-the-box you basic bar you know falls of docker that when they first started follow up the 12 factor application model so you know everything gets long standard out so this does the same thing everything gets a lot of standard out but on top of that you get a note level blocking which takes standard out standard error and applies that at a cluster so there's many different things you could do with logging in the queue

you can log your single application if you want you can log the entire cluster you can log just one container if you wanted it's pretty much up to you the way you design it what I do and what we do at work in production we you know daemon sets they're called which is just a container that runs on every node and it logs and sends everything is blocked so we just get all of our data collected all the time and moved off the nodes so if anything happens to number of the container goes to another node we still have all the logs but like I said before with how the abstraction happens for the container daemon same thing happens here

so that's also extracted and you get a log of API so you can put whatever blog you drive what you want or do using your enterprise insider so you Splunk out syslog there's I can't less I my book I think there was like 15 or 20 of them available just right out of the box that you can start using to do all of your login that's all JSON so little insider tip if you get lost each message one line at a time keep that in mind if you have staff traces then there's this new guy out there called the admission controller this is new as I think 112 of kubernetes I believe so this sits between the API server and the

cubelet that is responsible for running your containers and it can do even more security within your cluster you know you can tell it these are just some examples of control that is available but it's customizable and when you put whatever you want in that admission controllers have limit anything that goes on so you can tell like always pull images instead of if the image is already pulled out on the node and you container guys it has to come back up you can tell the admission controller all right don't use the image stored on disk always pull it no matter what you know that's just an example of what this can do I gave it through like great rate

limiting it get deny escalating exact which is pretty cool because we had developers at one point figure out that our robot accounts were able to exact into the container and they just would put all the logs in there Jenkins bill dog story if we're looking like the bill logic on man what this 25 pages of bill blocks right in here then we love her like Abba jankins a robot account is able to exact into the container and they just told it to you know show us all the logs so things like this you tell them like here I can't do that you can also do that far back but this makes a little more granular our task versus

space so now we're going to talk about workloads who doesn't love Yammer alright like kubernetes is a pretty awesome ecosystem but like it might it's just hard to wrap my head around the fact there's like a large scale mr. lead computing and violent available it's completely written down everything in this world is gambling so even if you define nodes in in compute nodes and workloads within the cluster that's even in the animal everything is yeah it's translate the JSON obviously they kinda it's all defined in yeah mole but there's a lot of really good utilities there and you can statically analyze line works so basically you know it's part of your pipeline as part of your

builds throw some static analyzers in there some code anything inside your pipeline that looks at the animal you know tell it if it finds certain strands allow this bill to exceed it's a pretty nice sort of nice thing about having animal the XD cluster talked about that beginning how you secure the sed cluster so basically when careers deployed sometimes you could deploy a CD alongside of all of your work or nodes or the master nodes don't do that deploy the NCD cluster independent of that and keep it isolated fire all depend between their workers master and the SV knows everything an se D is stored clear text so all that API traffic is pretty much quick text so you

want to encapsulate that TLS don't let your service expire so you know in Krypton oppress and use the primitives like I talked about before because there's some obfuscation that you can do like they have like the base64 encoding of secrets which you know like basics for is secure but there's stuff you can do with EDD always make sure that you're on ESET v3 use TLS encrypt everything at rest limit direct post access you know same thing with using those job posts like nobody needs to log into a subpoena there's no need to log into those servers like if the cluster was designed correctly and it's being administered properly Exedy doesn't really need much kind of feeding you know just kind of

like you came around on a kubernetes job to be your backups we never have to touch yes to be if there's a limit that women host access all right suggestions best practices obviously you know keep your keeper clusters updated kubernetes has a ridiculous velocity of development like if you're going to look at the change logs when they release one like one eight four two one nine oh it's like nine pages of changes like and some of those are really important so make sure you read your change logs before you do the piece no young update that's why TLS everywhere obviously some you can deploy clusters with you want without TLS but the kubernetes api has a very easy

TLS implementation you know you just you can have a PKI bilbray pro right in the cluster and use it to do TLS so you can just pass some flags to the cubelet you have a secure cluster use the CIS benchmarks he is benchmark nothing last time i checked for kubernetes like eighty five pages long it's it's a boring try to read but there's this really cool utility by aqua cycle cube bench you can run it runs the container and it will analyze your whole cluster and and it's the whole utility is based off of the CIS benchmark so it'll just output all the stuff on how you score against the CIS benchmarks pretty nice it just tells you you know

you can see here says you know privileged exact is fail because you so that's a very nice utility to use a kitchen cost you to check out best practices can help you secure it without even having a bunch of dots limits you call access so cute colors the command-line utility that interfaces with the API it's basically like SSH for this sort of the cluster you know with limit the access to that because you know it's just like I said the beginning like sudo can be nice but you know pseudo - ah I was like no counting or anything else is not do it so do this keep the same thing of mine where the cube Caudill will make sure the access

is restricted and everything all those actions are logged obviously a lot of make container image builds there the cluster that we brought in production does not have the docker full the full docker API running and it has container these so we came and build images on the clusters at the guava - so we do all over building outside of the cluster using build servers you know during that process static analyzers are run and they could analyze everything that's going on in there do integration testing you can do any type of testing that's necessary to keep the best practice followed within your organization so you know that what you're deploying is secure or not you know just some image

from docker hub that has a Bitcoin miner in it escape it as Ubuntu 16 that happened actually don't run processes or hardly anything these brew right like even if you're running a web server in a traditional world it has to run through it because it's ramble over for 1024 in this world you don't have to run anything on that port leave on a service that talks to port 80 but a different port on the back end so all of your services can run has gone users I think we have like maybe one or two potatoes that are on his room and that's just they have to in a way but 90 something percent of all the images in

our environment number one really deserve to insist on necessary utilize the community security primitives that I mentioned there is a ton of them the documentation is amazing I was on sick docks for a long part of the today's project help working on the docks there are so many dogs out there and the kubernetes docks are good you don't have to do a little bit of medium or any of the other places like just go to commence die of the docks there they're absolutely phenomenal you know obviously do least privilege because as you can very easily in this world you know you can add a couple of lines of the animal that implements least privilege in your security and your

security configs them pretty good to go there there's some resources here anxiety is benchmark link that's a good one cube sack is good too there's a lot of API stuff on that page and help you figure out the way you want to secure it I told by Gareth I worked with docker it's called a queue test all that does is he analyzes email they across it through the API server just says if your exam was good enough there's something here was something that that's a good utility audit our back I wish I had that last weekend I just found during the research for this presentation on into our back we'll take all of your our back rules and and draw

you a nice picture what's and then the Aqua SEC security company they've been there I think they're mostly like document detainer security to remember it's a security vendor that's free it's just cute bench which is analyzes it's the best practice analyzer it's all of us it just reads through doesn't make any changes it just routes through all of your deployments and everything it tells you if you're following CIS benchmarks last link is the network policy recipes there and github there's like 40 up I believe last time I checked at a medic has written it covers pretty much everything with really nice diagrams that show all the workflows and

if you know we don't fool time work on human is but there's I think it was pretty much that Karen feed the cluster it doesn't have that much like if it does it's because there's operator and misconfigurations that's basically it other than that it's pretty much it pretty it doesn't read the scheduler within the environment does a very good job of managing itself second question is like for running jobs do you guys ask after that kind of probable and look over or do you let applications I mean we said don't give access to write the developers do not nobody gets cucumber the only thing that has that is the build robots the developers are given

they could deploy pretty much whatever apps they want like and they are it's all namespace and isolated and their resource constrained the resource reporters like we saw here but we we use what's called their helmet arts I'm not sure if you we use Helms so we have a definition of applications that are allowed to be deployed and vetted and tested and then we have the help charts that are developed an age of state hold on the helm charts and for the developer the only thing they have is a pretty much a key-value answer so like they have a values file it just says this version this environment this app they have predefined deployments and you can add a

little security that's in yeah a lot of it is like not awful questions so a lot of it like there's abstract now you still manage it within the yamo files like I showed earlier so you'll have your definition of what what your deployment is and you can have your environment variables will be set in there but you can manage that as a secret and that gets stored in its suv's the key value store so you don't actually do much interfacing with that CD yet she kind of just does its own thing on the background story yes it's just a pointer that says this is the secret that's deployed in the cluster and you can use

whatever you want to manage those secrets like a Shakur's ball or any other storage mechanism