Zero Trust in the Matrix: Hardening Kubernetes for the AI Frontier

I guess it's four. I'll get started. Uh good evening everyone. I'm Apur and today we are going to explore what I like to call the intersection of the chaos securing AI infrastructure on Kubernetes. As the last speaker, I know I hold a very important responsibility and that's to get you all out to the bar in the next 25 minutes. I promise I'll do that. Quick background about me. I'm a security engineer at Microsoft with uh research interest in Kubernetes, containers and application security. So here's the reality. Every company is now an AI company. If you're not already running LLMs in your Kubernetes, you will be soon. And look at this architecture. The user input comes in,

goes through an API gateway, talks to an LLM inference pod, maybe talks to a vector DB, and here's the interesting part. Tool execution AI today have access to loads of tools be it for code execution web fetching file access anything that you would want and as a security engineer that's where the real stuff is so our road map we'll cover uh why the threat model is different look at the major attack patterns that I've seen in the wild and then shift to practical defenses that you can implement I leave you with actionable items for the Monday morning So this is the key insight, the threat model shift. In traditional app security, you control the validation layer. User input goes

through your code before it does anything. But with AI apps, the LLM is in the middle and you don't really control what is being passed through the tools, what's being passed through the kernel. It's unpredictable by design. Prompt injection means user input can effectively become code and you are basically just talking in English for it to execute code. So moving on to our first attack pattern and that's sandbox escapes. Ever asked your favorite LLM to analyze a CSV because you are too lazy to do it? Try following its thinking traces and you will see LLM pulls up a code exec interface. That's the sandbox. The promise it's at isolation. However, the reality might be different.

We have seen sandboxes running as root because they needed to install PEP packages or maybe because the pod died and they had to restart it in privilege mode. Internal services accessible on local host, no egress filtering whatsoever and the cloud metadata service wide open. This pattern is shockingly common. It looks simple, but we have seen it. The sandbox is actually giving you a false sense of security. So here's what the attack actually looks like in practice. An attacker crafts a prompt that gets the LLM to run this exact code. We aren't looking for a complex buffer overflow here. We are just using the tools that we built. In here we see the attacker can exfiltrate

HC password or any environment variable directly to their C2 server. Even worse, they can hit IMDS service at 169254 IP that is there to harvest the cloud credentials. If that works, they just don't have they don't just have your container, they also have your cloud account and there's no exploit needed really. You just needed to ask it nicely. From a Kubernetes perspective, this is where the gap becomes a chasm. On the left, you see what the security team thinks that you deployed. The textbook secure pod manifest. It's nonroot. The file system is locked down. Egress is denied. It's perfect on paper. But look on the right side because many AI frameworks and sandboxes need to perform

tasks like pep install at runtime or if you attended cat talk yesterday pods there and you have to restart the pods and often times during debugging you will start that in a privilege mode that's a risk. Uh when this happens the uh the pod itself might often default back to the root access mode. Developers might also skip network policy because they are facing some issues while debugging. The result is that your secure sandbox is actually running with full root privileges, unrestricted network access, and a direct line to your cloud metadata server. The one thing that I want you to take from this slide is audit what's actually running, not just what you intended to run, and

you'll be golden for a lot of attacks that we are talking about. Moving on to the next major pattern. It's our favorite OASP kit. SSRF attack pattern two being SSRF through vision APIs. Describe the image at this URL. Sounds innocent, right? But what happens? The URL gets fetched serverside from the cloud provider's infrastructure. And what if the URL points to IMDS IP? This is SSRF as a service. You're literally asking the AI provider to make serverside requests for you. and many of them will happily do. Now I'll be honest there are block lists and you might not be able to reach it directly but prior doing a redirect a 307 or 302 redirect there's the talk to

problem time of check versus time of use they validate the URL at request time but the actual fetch happens much later in between the URL does a 307 redirect to an internal IP maybe IMDS IP and legitite.com is actually loaded with IMDS credentials that's because the fetcher is flowing the redirect uh through a different code path than what's actually doing the validation. Takeaway here is revalidate after every redirect and that will also protect you from a lot of these redirect type of SSRF attacks. Now on to the third major pattern and this one's uh cross tenant isolation failures. Consumer AI platforms are multi-user by nature. Your code is running next to someone else's code on

shared infrastructure. And here's what breaks. Predictable IDs, internal APIs that trust anything, session state that doesn't know whose session it is. The moment you can list someone else's session and execute in their context, that's the whole ball game. In Kubernetes terms, network policy works great for pod isolation. A can't directly talk to tenant B. But what about shared services? the database, the cache, the internal APIs, even the internal bootstrapping code uh that's used for your code uh sandbox. They trust the traffic because it's internal but there's no tenant context which is flowing through which means the service layer is actually the weakest layer. You need identity aware access not just network positions. Okay, let's shift to the defenses. Five

principles for zero trust with AI workloads. One, never trust LLM output. Two, sandbox doesn't mean secure. If it just says sandbox, add real isolation. Three, eress filtering is mandatory, not optional. Four, block IMDS, period. And five, tenant context must flow through or be provided to the internal APIs. These aren't just theoretical. They directly counter the attack that we have just discovered. Okay. Uh moving on the first defense uh network policy for egress. So that YAML that you see here that blocks the metadata service by IP blocks all RFC1918 private ranges and only allows HTTPS to the actually oh that one uh yeah the key here is the accept block. Explicitly deny internal ranges. Don't rely on allow what you need. Also deny

what you definitely don't need. This alone I think would have stopped so many of the attack patterns that we have seen with multiple providers. Port security context run as root is non-negotiable. If you need package installation, use pre-built images with everything baked in. Read only root file systems. Prevent persistence. Drop capabilities reduce the attack surface. sec profile runtime default block strys calls these are table stakes if your AI sandboxes don't have it fix it soon and finally the standard containers versus the kata debate uh standard containers share the host kernel that's a massive attack surface one kernel exploit and you are already on the host kata containers or gisor containers take a different approach they run your

workloads in a lightweight VM with its guest kern kernel there's real hardware enforced uh VM boundary even if an attacker gets kernel level access they are in the guest kernel not the host they'd need a hypervisor escape which is much much difficult to target Azure uses this uh for confidential containers the performance and pricing overhead is slightly higher than traditional containers but for code execution sandboxes uh which are handling untrusted AI input or handling sensitive data that trade-off is definitely worth it. If you are on Azure, look for confidential containers. If you're on prem, Kata is open source and integrates well with containerd. For SSRF defense, here's the pattern. Pass the URL, resolve DNS, check the IP,

and then uh do your own validation. The key here is that while loop that we see, every time you hit a redirect, you reissolve DNS and recheck the IP. that defeats the talk to bypass. Don't use allow redirects equal to true blindly and you're golden. Handle redirects manually with validation at each hop. And talking about IMDS hardening for AWS, require IMDS v2 tokens. Maybe set hop limit to one that blocks containers from reaching it since the token request won't actually go beyond one network hop. For GCP and Azure, use workload identity. Your pods authenticate with their own identity, not the node service account, and that eliminates the need for IMDS entirely. Workload identity is the right answer. IMDS is a legacy

pattern that shouldn't exist in modern deployments, especially the AI deployments. Detection matters. These are the signals to definitely alert on any to IMDS IP, DNS lookups to metadata URIs, outbound traffic to nonallow listed IPs, cross namespace traffic that shouldn't exist. If you are on Azure, Defender for Containers has a built-in alert for metadata server access. For deeper runtime diction though, I love Falco. It's open source, Kubernetes native, and you can write any custom rules for exactly these types of scenarios. Okay, so what do we do Monday morning? Immediately audit your egress what your AI pods can actually reach. Block IMDS with network policy and IMDS v2. Enable workload identity and check if your sandbox are running as rule. Add falco

rules. That's a cherry on the top. For longer term uh evaluate gisor or kata for code execution workloads and consider service mesh for tenant isolation with MTLS. These are all actionable. Start with the audit. Know your current state and go from there. Four things to remember. One, AI workloads need new threat models. The LLM is untrusted. Tools extend your attack surface in ways traditional apps don't. Two, sandbox might not always work as intended. Ensure through audits that sandbox configuration is set to isolated. Three, SSRF via AI is already building upon traditional SSRF attack vectors. vision APIs, web fetching, tool users, all of these are SS uh all of these are SSRF vectors. Revalidate after each

redirect and four multi-tenency is genuinely hard. Tenant context must flow through each layer, but shared services are where from an attacker's perspective all the golden eggs are. So protect that service layer. That's the talk. These patterns are based on active research. Specific findings are pending disclosure, but attack techniques are real and the defenses work. With that, we are open for questions.

>> How do you know you need the IMTS though? >> Oh yes. uh that's why like I think uh tenant isolation is definitely hard but at least we can add defense in depth layers right like we can start with egress filtering if we do need IMDS maybe switching to a workload identity or federated identity provider instead of direct IMDS line >> I think that's something but we need to work in layers so that IMDS the crown jewel is protected somehow yeah Hey, first of all, thank you. I was really happy to see this talk. >> Yeah, Leo was giving the blue team talk yesterday. So, we thought about this problem >> and um the URL one was actually really

interesting to me because um I agree that obviously we thought about that attack. >> Um and I was kind of wondering if there's a more Kubernetes online way to like give because basically people do like speak to internet they want and so just telling them no not very they will be unhappy if we tell them no. Uh and so we're trying to figure out how to balance you know giving them some flexibility but also lock it down >> ideally as web users that their own subten controls here. I was wondering if there's like a Kubernetes >> honestly very specific to Kubernetes I am not aware there might be the way I would approach it is like what the internet

access or what additional access is needed for then maybe go like restricted approach okay these IPs or these uh FQDN are reachable maybe redirect everything through a proxy. Uh I don't know. I uh I don't know specific to Kubernetes. Uh there might be but uh like these would be the general ones probably. Oh yes. Uh so I think uh so a lot of uh issues that I've seen and let me go back to that slide. Uh okay, it's not working. Uh I'll just answer uh I I I couldn't go to that slide. But uh what I mean by tenant context is I'll just leave it. Uh so a lot of uh scenarios that I've seen are

not because the tenant context was flowing through the upper layer or the application layer that was shown to me. It was because of these shared services which were running. Uh maybe one was summitting jobs. Uh or uh I don't know if you have used E2B. uh E2B is the new code execution uh interface that the AI providers are using and as part of E2B there's the shared service which is used to initialize the job list the job maybe access files from other uh mounts and everything so it that shared service doesn't have any concept of uh who the user is or what the context is and since that is lost in that in the translation

there I that IOR attack becomes very much possible in that instance and uh we go from there like trying to exploit those shared services. Awesome. I think with that we'll end the presentation. Thank you folks.