School of /Proc

back folks here we go it's like like conference over that's it oh my gosh all right um I guess uh yeah I guess we can get started and then I will drag people inside slowly whether they like it or not um all right our next presenter today is Alexander wise who's a technical account manager at verica and his talk is titled School of proc kubernetes security and the post PSP world just give it up foreign thank you uh yeah like every other speaker has said so so cool to be back in the conference uh setting in person seeing people I love this uh but I'm I'm Alex I'm here to talk about kubernetes security I'm here to talk about the the recent past and near future of kubernetes security uh so about a year ago back April 2021 uh with uh the release of kubernetes 1.21 the kubernetes team also announced the deprecation of pod security policy API and kubernetes pod security policies were the only built-in way to configure your kubernetes cluster so that uh a workload could not be submitted that that could trivially escalate to root on the Node um but but kubernetes deprecation is just an initial first step it's initial early warning that says hey this is going to go away eventually um you know start making plans to come off of it find something else that will work for you and there really are no plans to remove pod security policy until kubernetes 1.25 which is the next kubernetes release 1.24 came out last month um so the next uh the next version of kubernetes probably will not have pod security policies in them um there's still no reason to worry if you if you might be using pod security policies because you have most most folks are not on the freshest Upstream release of kubernetes as it comes out and you will have at least a year of 1.24 being supported before that eventually goes away so you're not going to be forced to force off of pod security polies a lot of folks are not on are not using pod security policies it's a we'll get into it it's a little bit difficult of a of a tool to use um but yeah we will uh we will get into that later um a little bit about me um I am my name is Alex wise um I've been doing kubernetes stuff for a little while my first production cluster was uh 1.6 a little while ago I've since then secured kubernetes workloads a lot of different places I've done for SAS for robotics for ML workloads um if you want to ever nerd out about development environments or build tooling and how to secure that let me know I'm currently at a company called verica we build tools to help you secure and find the the safety margins of your kubernetes clusters of your Kafka clusters we also do work around town for the software Freedom School I teach for them write some classes for them helping beginners learn how to use why to choose open source software so some folks may have seen me for that my you can find me on Twitter AWS Architects so that's me I do want to lead off here with a couple quotes from the co-chairs of Sig security for kubernetes the first one um indicates what what value pod security policies gave us uh what how people used them what was the threat model they were trying to stop um the other one we'll talk about the design decisions of philosophy where they want to move kubernetes to why they're deprecating pot security policies and what the replacement needs to do better so the first quote here is from Tabby Sable from her talk from kubecon last year PSP replacement past present and future permission to create pod should not mean permission to be root on every node in the cluster so this is the problem that pod security policy existed to solve kubernetes is a is a system administration tool that wants to do system administration things it wants to help you run privileged workloads perform upgrades things like that so if you're able to run those kind of privileged workloads there are a thousand different Paths of attacks ways to to escalate your privileges out of a container way to perform container breakouts that mean that there's really a very fuzzy line between create pod and kubernetes and root on the Node and we'll talk a few of those in a little bit but but that's that's what PSP's solved that's what the replacement needs to solve that's why we're talking about this the next video is from Ian the other co-chair of Sig security Tweet back in 2020 saying kubernetes emphatically not secure by default um and and we should not paper over that fact that that what kubernetes needs to do better is to be secure by default um you shouldn't need to be a secure container security Guru to to be able to to run a secure cluster so that's where we see PSPs we're falling down and what The Replacements May hopefully do a little bit better it's really important I'm going to put it here we'll probably refer back to it but kubernetes needs to be secure by default so wayfinding slide here we just finished intros we're going to talk about what is slash was the Pod security policy we're going to dive a little bit more into what those rough edges were why it's going away going to talk about pod security admission it's replacement um and then if pod security admission doesn't work for you or if you're if you're a security research or an attacker out there trying to see what you might be up against a in a cluster we'll talk about some of the other security controls that solve this problem in kubernetes all right so diving into it what is a pod security policy anyway so pod security policy was an admission control plug-in it was it's the only uh security system like this that's built into kubernetes natively and that that's important right it was it's pretty long in the tooth it was G8 in 1.6 and what it does is it lets you configure requirements for the security context for any pods that run there and those are extremely granular we'll take a look at those some of those but you can be very specific about the secure the security controls that you want on your uh on your workloads it's also extremely flexible you can configure your pod security policy to to default you to secure settings to to to secure or just validate and say no you're trying to run as privileged we won't let that on the cluster um you can set these policies up so they apply to to all namespaces and everything cluster wide or you can scope them to specific privileged privileged name spaces so there's it's very flexible and very granular which which creates some foot guns for for folks um who may not have expertise in this so uh this is some of the granularity this is just a sampling of some of the the controls you would have uh with pod security policies setting those security contacts you can see things like running privileged containers um you can see in Linux capabilities any sorts of of volume mounts you might be able to add there's um SC Linux modes and app app armor settings that would be configurable here um and if you see there's a an allow privilege escalation Boolean which is kind of my favorite Boolean like set this to false and hey we've solved security right we we did it we fixed cyber um yeah maybe um anyway so how does this work in practice so uh so pod security policies that they build on the the existing R back and workload identity apis of kubernetes essentially every pod has a service account associated with it and and that says what permissions that pod has so PSPs just sort of build on that and say okay if you if you have this identity apply this policy to it you might have these extra permissions or you might be running you know very very restrictive but pods map to service accounts map to either cluster roles or roles and then those map to PSPs if you're if you're mutating you're going to get one PSP if you're if you're validating you might get a whole array of them uh the drawback here is that if you're not using workload identity in um in your kubernetes clusters so for instance if all of your pods run as the default underscore default service account in a namespace you might have some heavy lifting to do before you could start to use pod security policies but in theory what I could do here I could make a a really restrictive cluster role that is cluster-wide that everything sort of gets by default um and then if I have more privileged workloads if I have security tooling if I have I have operational tooling or or Legacy workloads then I have a role that's specified to a single namespace and gives me just those perms there so it can kind of do this high this really cool hierarchical uh setup where I I can get my least privilege right so that's what this kind of looks like here on the left we have a cluster role that's cluster wide it's very restrictive on the bottom there that says resource names that is the name of your pod security policy the block all the things policy on the right and your lift and shift Legacy namespace you've gotten all the perms pod security policy so that you can kind of do this this really cool thing where you have have hierarchies and you can run the things that need permissions um just sort of as with those elevated permissions without making the cluster as a whole um less secure pretty cool so I think I can do something like that how do I turn it on all right so PSP uh you would turn it on just by by passing a flag in this enable admission plugins flag um to your the cube API server start command um you'll actually just be a big long string of comma separated plugins that are that are also on by default this will typically live in your cluster build out scripts or your your infrastructures code that kind of automation um so I I can do that I can update this right and the idea is I'll turn this on and then I'll go in I'll start looking at my workloads I'll start crafting policies uh similar to what that previous slide was and and set this up right and so I'll do that and everything breaks because uh pod security policy is a security control it's denied by default if I turn it on and I don't have my PSPs created uh no pods can be scheduled everything gets denied everything gets shut down and everyone's mad at me um so I need to know I I need to know exactly what my policies are going to look like before I turn it on but I I can't turn it on until I know what my policies look I kind of I kind of have this chicken and egg thing right I I can't quite uh uh I don't know a priori what my PSPs need to look like and I I can't really find my way into them but what I could do is I could like make a really permissive Global policy then turn it on and then start incrementally um you know creating policies for different workloads and finding these privilege I could do that there are some caveats um you know it will take some some trial and error to get this right probably depending on how complex I need to be with it and it's not a great default I mean we can just say that right this is this is this is uh from a from a defender's perspective um I I have to do a lot of maneuvering to get these turned on to get these running in my environment and and if we're trying to be secure by default that we're kind of falling down here all right one moment cool any questions so far in that nope all right so we're going to dive in a little bit deeper into pod security policies um we're gonna say okay I have done everything I said in that previous section right I've set up these these cluster roles I've got everything running I think I'm running pretty secure fairly least privileged I'm feeling good about it I'm probably in the top 10 percent of secure kubernetes clusters operators because a lot of folks don't use PSPs because they're kind of difficult then I come across a blog post and it says kubernetes insecure host path mounts okay okay um I dive in I dive into this blog post and I think oh duh right cool so kubernetes lets you mount a server's direct a directory from the server inside your container as a as a volume inside that container similar to how you would do with Docker right well if I just Mount the root directory of my server inside my container I can do all kinds of shenanigans uh this blog post has you just rooting into into that directory and taking over the server that way but you could you could cut Etsy Shadow you could steal SSH Keys you could steal the the kubernetes node search right there's a lot of Mischief you can have if you have full access to the root directory on a Linux server but I can fix this right I can block this PSPs I have them set up they give me the tools to to fix this and it would look something like this right I don't I don't even I'm in the cloud right I don't need this host path I don't need to be mounting directories inside of containers from from my server my my servers are mostly immutable there's no there's no uh State saved there um so yeah I'll just I'll just I won't allow um I won't allow host paths I will update all my pod security policies on my cluster and say you know these this is the list of the types of volumes we allow these are sort of kubernetes native things are Cloud native things we don't we don't need host paths we don't want to touch them and I feel good I feel super secure [Music] um until I see this tweet from Duffy and that's a that's a heck of a tweet uh this can run this on any any cluster that's not running pod security policies and get get root on a node and it's a heck of a tweet right um but I I'm a container guy I'm a kubernetes guy I can I can so I can parse this right there's nothing here that I can't parse so Coupe GTL run pretty standard the image doesn't matter because we're overriding it okay host PID true inner Center all right I think I see what's going on here and I I I remember something I heard about about Linux containers so Linux so containers aren't little boxes right I remember Alice goldfuss's talk here and she had a mantra about Linux containers containers are processes anchored to namespaces controlled by c groups uh so so container is just a process it's got It's got a pids uh it's in a namespace and that says what uh the Linux namespace says what other processes it can see and what what devices it can see and the c groups control what resources has access to so if I do a Docker run okay I started Docker container yeah I can see that it it has a a it is a process just on my server it has a PID um I can do some Linux internal stuff and see the namespace that it's running in and I can use that NS enter command and go into that namespace um you know I can also see this information about the C group it's in so so containers are just processes and namespaces so going back to Duffy's tweet all right so running host PID true that means that I'm running in the hosts process space the host name space and I'm using NS enter to to go into that namespace that means that I can see every process and every device on the Node I can touch every file so that I'm rude at that point um everything in Linux is a file so if I do that I am essentially root on the server but I can fix that right I can I can I can I can use my PSPs to make host bid false and and privileged false and host Network false and privilege escalation false I can do all that to stop it but it's not very secure by default I've had to be a container breakout expert to understand these Paths of attack so that I could craft these policies so that I could stop it and and that is not a place that kubernetes wants to be um so wouldn't it be cool if kubernetes bundled up everything that it knew about container escalations and privilege escalation and container breakouts into a policy right because I talked about insecure host path mounts and host bid takeovers but but you there's mounting the the container runtime socket inside the container there is running with cap BPF and capnet raw to to forged packets there there is uh unmasking the slash proc Mount there are all these sorts of container breakouts all these acrobatics you can do to get out of a container when you can run privileged workloads and wouldn't it be cool if kubernetes bundled all those up for me so now we get to the PSA on PSAs pod security admission is the replacement for for pod security policies it is also an admission controller uh entered beta very recently uh 1.23 still behind a feature gate pod security admission is is significantly simpler and then pod security policy uh it it works at the namespace level you essentially uh in kubernetes labels are key values that you that you assigned objects so you're assigning these to the namespace object and kubernetes and basically setting it to one of three security levels and any any workloads any pods that run in that namespace will need to conform to whatever that that that level is um so what you're getting here you're getting a loss of that granularity you don't have that that tool kit that you had um that you could use to craft really specific uh policies but you get better defaults if you're if you're not a container security expert you you can um get a default secure setting and reasonably trust that that you're not there aren't uh Paths of attack that there aren't privilege escalations that could be performed so I like this um this first sentence from the Pod security admission Docs uh kubernetes pod security standards Define different isolation levels for pods these standards let you define how you want to restrict behavior of pods in a clear consistent fashion security restrictions are applied to the namespace level when pods are created so this is all about pod security standards so let's let's dive into what those are I said they were there were three levels right the first privileged basically no restrictions right if we're not going to stop every privilege escalation um you know we will assume that the privilege can be escalated there let's let's not restrict anything Baseline is your your trade-off between compatibility and and and this is key here the known privilege escalation paths um it it wants to allow as much as possible to run without opening the Avenue for create pod to be root on the Node um some some of the things that are enforced in a baseline you can't have privilege true there's a list of specific Linux capabilities that you can have but none of the really good ones like capsys admin um no host paths at all we saw host paths were really problematic um and there's there's a list of others but those are those are some of them the the Third uh mode that we can run in here is restricted um and that's that's enforcing every known pod security best practice so things like your containers can't be running as root um any capabilities have to be have to be dropped any learning Escape abilities uh whatever the the host SC Linux mode is has to be enforced things like that um also there's one where where no child processes that are spawned inside the container can have more permissions than the container at the the parent PID things like that so so those are our baselines um very simple buckets that we can we can put our workloads in and it's behind the future gate right now but it's enabled by default we are moving closer to secure by default and kubernetes with pod security admission all right so let's let's take a look here what do these labels look like I said it's just two labels that you would set on on the namespace object to set this up so the first one is this mode label um so mode you enforce audit or warn the only difference between audit and Warren is audit is put writing to your audit logs um and Warren is just writing out to the console um and then you would the level is one of those three buckets the privileged Baseline are restricted so you would say something like pod security.kubernetes.io enforce Baseline and in that namespace now every pod that that could be scheduled in that namespace must comply with the Baseline security standard so no running is privileged uh you'll you'll get rejected you won't be allowed to run in the cluster there th