From Pods to PCI: Translating Kubernetes Security for Security Audit & Compliance

Um, hi everyone. Um, good morning. Thank you so much for being here. Uh, my name is Udo. I'm a security engineer. I live in Seattle actually. Um, I've been working in the field for about 5 years and I want to say thank you to all for being here to listen to my talk and also thank you to the field to the B size Philadelphia team for making this possible. So just as you can see my talk from pause to PCI translating Kubernetes security for security audit and compliance. I have to start off by apologizing. Um this is not a talk about Kubernetes security specifically. I don't know I assumed everyone was familiar with Kubernetes security maybe

because I work there but basically while working I've noticed that there's a lot of miscommunication between the Kubernetes teams and the security audit and compliance teams trying to execute audits especially in cloud native environments and the point of this talk is to kind of like help both teams understand each other and sort of like achieve at a achieve a shared and common objective. So and even though this talk talks about like PCI um it's not restricted to the PCI compliance framework um it can be applied to other security compliance frameworks like so 2 fair um so just um putting that for now. >> Okay. So um why this talk matters uh just like I mentioned um on one hand we

have Kubernetes engineers on the other hand we have sorry Okay, we have security compliance and then we have security audit. Um, all the teams are trying to I mean the security audit team is trying to like obviously achieve security audits and they're asking the Kubernetes compliance team for evidence but both teams don't really understand each other and so there needs to be like a middle ground for both teams to be able to communicate. So just looking at the example um up here um so this is a Kubernetes security context which basically says that a Kubernetes port should not be able to run as roots and that um the capabilities for any user should be restricted.

Sorry um that sort of threw me off. um that the capabilities for every user should be restricted but when a an auditor is asking so tell me how you are implementing this privilege in your environment they might not really understand this if you like showed them the screenshots even though basically this effectively implements this privilege in a Kubernetes environment so basically trying to get both sides to see each other and understand each other that's what this talk is about So um basically um if you look at this table like I mentioned you know if you're familiar with Kubernetes or familiar with things like port security network security [snorts] um compliance auditors are thinking about how you implementing

access control how you implementing network segmentation um both sides might not know that they're speaking about the same thing. So basically this table is sort of giving you an understanding of how different PCI sub fedraph requirements are being met in Kubernetes control even though they might not really apply to like the native language of like data centers, firewalls that are more familiar to the traditional security compliance professional. Um more examples here. So just like I said because both sides work in different fields and they don't really always understand what each other is saying it's best to meet at the middle ground and what exactly does that mean that rather than saying oh how are you

meeting PCI6 or oh we have our back for implementing AM both teams can come together to talk about security objectives so I feel that you know the collaboration between both teams will be more productive when you're talking about security audit compliance in terms of like access control, change management, audit logging, vulnerability management. These are things that both sides can really talk about and understand and sort of like bridge that understanding gap between both teams.

So hopefully by the end of this talk um the takeaway of this talk should be that you should be able to apply this mappings when you get back to your offices especially when it's like if you have an upcoming so two audit or an upcoming PI GSS audit this should be a framework you're able to apply so that both sides of the aisle are able to understand each other and achieve the right um security audit in the cloud native environment. So I'm just going to start digging into different security controls and talking about how they map to Kubernetes. So right here uh we have access control in Kubernetes. When it comes to Kubernetes, how do we achieve access

control? So we have rules and cluster rules. A role is basically what a pod is allowed to do. A cluster role is what is allowed to do across an entire um Kubernetes cluster. And then role binding basically binds um a role to a pod just so that you know kind of like um this privilege in a more traditional environment. What is a pod allowed to do that is defined by the role that it is bound to and also service accounts in terms of the Kubernetes workloads. These are the identities that Kubernetes workloads have. um you know in the Kubernetes cluster uh we have like namespace uh you know when you talk about logical isolation separation of GTS

um you know things like network security separating boundaries boundary controls that translates in Kubernetes in terms of like namespace and network policies um when you think about things like change control how is change management managed in a Kubernetes cluster we talk about things like admission controllers and in terms of like audit and logging which is obviously like the central issue when it comes to security compliance and auditing we can think about Kubernetes API server logs. So just to show um say you know you have an auditor come in and say oh you have the service satisfying in a kubernetes cluster we need to get um security audit evidence in terms of like how are you

implementing identity and access management these are some of the evidence types that you could provide in terms of like you know what is the outback yo snippets um you know what are the output of cubectl um thinking about things like opa gatekeeper you know how was those policies first and audit logs. So just to show an example of this um you know in real life. So here we have created a service account. We're calling that the audit reader and just so you know this shows like you know the configurations of that service account and then um sorry and then you know that is the role binding. So basically um trying out like you know running these ports um using

this service account role bound to it and trying to like get secrets especially when we know that the service account lacks the permission to get secrets we can see that that is an error um you know like that that doesn't work. So that is something that in terms of like you know if an auditor is asking okay what is identity and access management how are you enforcing identity and access management you know this privilege in your Kubernetes cluster this is an example of an evidence or like you know what you can think through in order to demonstrate compliance um towards that compliance objective and then moving on um the next security control is network segmentation. So

basically this is talking about you know obviously in traditional security we have firewalls we have VPCs you know um sensitive workloads should not be able to communicate with nonsensitive workloads this is a very core requirement of PCIDSS and basically this is how it's implemented in Kubernetes and we do that using network policies and name spaces so these are some of the evidence you can provide say if an auditor is asking so how exactly is network segmentation you know and isolation being implemented in Kubernetes clusters. Um we have like the network policy yam snippets. We have like the cubectl output of network policies. We can also test you know um we can like test interconnectivity between various boards and um screenshot

that as evidence for something that can be applied towards the security audits. Um yeah here we have more of that audit trails logs name space boundaries and just so this table is basically like helping you understand how exactly the various PCIs requirements map to actual Kubernetes features. Um this is kind of like a framework you can use when you're thinking about okay how exactly do we show that we are complying with the network segmentation requirements of PCSS so two fed ramp um these are some of the things you can think of in terms of like working with security auditors to meet those objectives and then just to show another view world example. So for example here we have

uh [snorts] so okay so yeah we have like two ports in the Kubernetes cluster by default without any like security implementation these two ports can communicate with each other and obviously that goes against um security compliance framework requirements and then when we apply um a network security policy we see that that connection is blocked. So this is another example of an evidence that can be provided towards um the security compliance auditors. Then moving on to port security. Basically we're trying to say you know when we think about a pod in a Kubernetes cluster what are the security capabilities that those ports can perform for example can they run as roots? Um what permissions are being

assigned to them? um you know what um elements within the Kubernetes cluster are they able to communicate with and all this is sort of like collocated in the configuration that is applied to a specific Kubernetes code. So again obviously um just again just like we've spoken before um you know what are some of the evidence you can provide in terms of like when a security auditor is asking okay how exactly is security port security being enforced in the kubernetes cluster um we have the port security admission configuration we have audit logs um we have like the specification of the ports basically you know what are the configurations that are being applied in kubernetes to control the security behavior of these

sports and those examples of like evidence that can be provided to the security auditor and just so again you know trying to place an understanding between the actual um compliance requirements and what that means in the actual Kubernetes cluster. Um this is like kind of like a framework that you can use to think about that. And then here we are seeing a real life um application of this. So basically um the PSS restricted is a restrictive port security policy. Um basically we are seeing like the labels here like you know what is prevented from doing being locked down. And then >> [snorts] >> um we are seeing that for example in this pod when we see things like um you

know this is a privileged pod so it allows escalation of privilege you know it basically has capabilities to do anything so there is no limiting of duties and it can run as the root user. Obviously this goes against security configuration best practices and then we see that when we run this port we get an error because it goes against our secure pod configuration. Um same here the error and then we see that when we explicitly provide good security configurations um you know not running at root not allowing privilege escalation limiting the capabilities of the pod then the pod is able to run properly. So these examples of evidence that we can provide to security auditors when asking about

you know how we well we have implemented security requirements in our Kubernetes cluster. So moving on um image security uh so basically this says that you know the image obviously Kubernetes is based on images that we deploy in clusters you know what sort of vulnerabilities do these images have are they vulnerability free are they clean what can they introduce to the vulnerability cluster this is a very important component of both Kubernetes security and Kubernetes security auditing and just looking at this tab we can see some of the evidence that points to show that we're using the right and secure images within our Kubernetes cluster. Uh we can see in terms of like the admission controller

um we can see fail deployment logs like for example when we um create secure admission policies that restrict unsafe images. When we run unsafe images, we're going to see evidence of that in the log and that is something we can um provide to auditors to say okay this is why this evidence of how we have enforced proper image scanning policies. um this as well. So just again another framework to understand how different individual um compliance requirements map to evidence types and what you can be thinking of when you are the person that's at the other end responsible for providing the evidence to meet um a specific security compliance audit. Um so just here another uh real world

example um you know being able to run a trusted image and an untrusted image. You know this is evidence that obviously we have um a clear security um policy running that blocks untrusted images. So this is an example of an artifact that can be provided uh for Kubernetes compliance.

I'm seeing here then moving on another important part of security compliance is change control basically who is allowed to apply changes are changes being tracked are changes being audited who has the permission to make changes that's a very core part of security that applies to both PCIDSS sock 2 and fed ramp um and then looking Here some of the things that we can think of are so um okay let me just speak a bit about cyeno and gatekeeper um opia gatekeeper and cyveno are tools that we can deploy in our kubernetes cluster that sort of act as gatekeepers in terms of like what actions are being allowed in a kubernetes cluster so for example say we

don't want anyone that is not an admin changing am roles in the kubernetes cluster we can create a gatekeeper policy that blocks that and that doesn't just serve obviously as a security control but that is also an evidence of a compliance um artifacts that we can produce to say uh we have implemented effective change control within our Kubernetes cluster and so that is a very good um example of that we can also look at our audit logs in terms of like what is rejected um if you're using like a githops um pipeline um you know you can also look at your logs in terms of like evidence of what has been rejected by the kubern

controller in the cluster. That gives a good idea of what change management looks like within the Kubernetes cluster. Um same here as before just mapping to various requirements that apply to change management. Um so here we have an example of an admission controller policy. In this example, for example, we don't want to use the host network and modifying roles is restricted and then when we run a bad um you know a port that sort of goes against these policies, we see that that is rejected. So that's an example of an evidence that we can provide during a security audit of a Kubernetes environment.

Uh so moving um into the final control. Um one would say that you know this might be the most important security control when it comes to security compliance auditing because auditing obviously is trying to understand what happens within the environment and how does that meet our security objectives and the primary source of data for that is in audit logging. You saw seen our audit logs. Um so basically you know what are some of the evidence that can point to this? We have our Kubernetes audit log that records everything that happens within the cluster. Uh we have our audit policies. Um we have like um obviously we can like collocate our evidence in our sim log

streams and not just about collecting evidence but also what is the retention of that evidence and like how can we rely in the integrity of that evidence and that is where log retention comes in and then looking at how that traces across the various PCSS compliance requirements and so coming to this stage. Um I know this has been um quite a lot. I know lots of people don't work in audit and compliance and this is a very niche um what would I call it now? A very niche talk focusing on people that are doing audit in a in a Kubernetes environment. But even if you're a security engineer, you don't necessarily work in audit. Someday you might be

tasked with like working with the security compliance team and helping them gather the documentation and evidence that they need to complete an audit. So these might be some of the things that you might be thinking of should that they come. Um so talking about some of the pitfalls in terms of this again like I mentioned you know people that are working in engineering don't understand what people in compliance are talking about. People that are in compliance don't understand what people in engineering are talking about. So both of them are not speaking the same language and there is a lot of misalignment which leads to a lot of inefficiencies and also talking about static snapshots traditionally in security auditing you

know you're bringing out like snapshots like things that were done in point in time but that is not how Kubernetes works it's ephemeral it's always changing it's always generating many locks per hour so that is something to think of in terms of as opposed to providing static evidence thinking about how that evidence can be continuously monitored, you know, for better audits and then ignoring the developer context. Again, this sort of speaks to the first topic in terms of like both sides not understanding each other. um you know when if a compliance person is coming in and asking to execute an audit in a Kubernetes environment but they don't really understand how the DevOps team works, how the software development life

cycle processes work, they're going to have a hard time really trying to get the evidence that they have. So there definitely needs to be a lot of empathy and understanding between those teams. Um obviously talking about logs again like I said a very key part of executing a successful security audit. Um lots of times these logs not exist because teams are focused a lot on like the operational velocity of the teams and not really thinking about observability and security compliance. So those are things that should definitely exist and should be used all the time. um you know not defined audit scope controls. Oh, okay. So, this is something that is really important like a lot of times

people could be like um you know um tell me how you have um say implemented network security in this Kubernetes environment and you know the engineer can be like oh we have network policies that's not a big deal and that's great like in terms of a security perspective but audit is about getting a testable evidence that will help the company meet its compliance objectives to still be in business which means that the security audit needs actual physical evidence of that security control. So that definitely needs to be emerging in terms of like a theoretical understanding that security has been implemented versus like the physical aspect of providing that evidence to show that that control has been met. And obviously again

compliance as an afterthought sort of um tying back into the first and the third topics um you know they should be brought into the part of the process. You know normally you know um you know people deploy services deploy applications and then whenever it's time for like sock to audit everybody's like running around trying to gather evidence but that is kind of counterproductive. The security compliance team should be part of the design and planning process right from the beginning of a Kubernetes design down to the end just so that they have visibility into how that system works and the audit process can be more efficient. So um what are the key takeaways from this? Um first of all you know again

having both teams understand each other you know not speaking about oh we have a network policy we have an admission controller what exactly does that mean in a security context what can both sides understand both sides can understand network security change management vulnerability management image security so that's a death better um context to talk within um mapping evidence to outcomes again just like I mentioned it's not enough to have security controls they have to be evidence they have to be attestable so that's something to work towards um preventive before detecting again lots of um you know kubernetes applications are collocated within the devops team which is not necessarily a security team so security usually comes in an

afterthought and so security should definitely be included in terms of prevention and when systems are being designed as opposed to like including them as an afterthought um auditor empathy matters. I mean I would just if I was going to summarize this entire talk in one thing is definitely going to be that point. Auditor empathy matters having both sides understand each other having both sides to work towards one goal. You know auditors are more management governance not very technical engineers are very technical don't have patience for the management governance side. Both teams need to work together and both teams need to understand each other and obviously compliance is continuous. It's not just one thing. It's not just oh this is suck to audit

time let's do that and move on. These are things that you should be continuous in terms of like you know do we have systems in place for gathering um evidence um do we have controls in place that will you know that we can discuss during audit time it kind of like I mean I know I'm sort of like differentiating security from audit but just the way we talk about you know dev secups embedding security into like the devops pipeline that security should also include like security audit how exactly would the security controls we've implemented play out in terms of like when an audit is needed. So just um a takeaway from this what can you take away from this talk? Um I know

that you know I'm not going to lie you know Kubernetes is like a very niche topic. Lots of people are hearing it for the first time but it's not something that you have to tackle all in one place. You can just start with one control um you know just and expand from there. um you know think about tools. You don't have to do this yourself. Kubernetes is a very complex and could be intimidating um platform. So think about you know some of the tools that you can use to make your work easier. And again you know I feel like I keep repeating it but empathy between those sides collaborating coming together understanding each other those are

definitely some things to think about and take away from this talk. So just in case um anyone wants to learn more about this, these are some like really good resources for like really understanding more about Kubernetes security. Again I understand this is a very niche topic but you know as companies I mean everyone is talking about AI now cloud native working on the cloud. These are things that are going to come in to your workplaces if they don't exist now. So these are some things to be thinking about just in case the time comes when you have to work in this area. So yes, that's it. Um, thank you so much. I know it was a little bit of a

niche topic, but thank you. Thank [applause] you.