Container Confidential: Secrets Management For Modern Applications

[Applause] Thank you. As we develop code faster than ever previously imaginable, we're leaking secrets faster than ever, too. 23 million hard-coded secrets were discoverable in public GitHub repos in 2024. Docker images being a crucial part of the modern infrastru cloud infrastructure are not immune to this problem either. 100,000 valid secrets can be found exposed in public docker hub images right now. The problem is only growing year on year. Between 2023 and 2024, there's a 2020 there's a 25% increase in the number of secrets exposed and discoverable. Leak secrets are the number one root cause of application breaches with 50% of all compromises being attributed to stolen credentials. That's not zero day vulnerabilities. It's not unpatched dependencies. It's

leaked secrets. If your team builds software, ships containers, or manages infrastructure, then this talk is for you. Whether you're already thinking about secrets management, or you're just starting to, this session that will help you level up your approach. By the end, you should walk away with a clearer understanding of the risks posed by hard-coded secrets and weak secrets hygiene. You should also leave with a mental framework for building more a more secure approach when managing your secrets in your applications. I'm Adam Matthews. I believe we've got the tools to help us fix this problem. I've been helping companies to improve their security posture through engineering and tooling for two decades. Leaked Secrets is one of the easiest

ways still to get breached. Uh it doesn't have to be that way. and I'm here to tell you that with a few smart changes, you can make sure that it doesn't happen to you. So, let's start with the basics. What exactly is a secret and why do we need them in software? A secret is any sensitive piece of data that an application uses to operate securely or talk to other systems. They allow your software to authenticate, to authorize, to encrypt and to securely interconnect. Modern applications are made up of many components. They have microservices, containers, serverless functions and databases all talking and inter interconnecting with each other. I'll ask you to imagine a simple application made up of just a front end

um web app and a backend database. Secrets are what makes the connection between those two components secure and private.

So in other words, you can say that secrets are the gatekeepers that controls who gets in and what gates and what stays safe. Without them, you shall not pass. They come in a variety of forms. A few common examples would include API keys which help you authenticate requests between services and also to access exterior cloud resources. for example, database credentials to for your for your for your front- end application to connect and interact with your data, your database. You've got encryption keys to encrypt and protect sensitive private data and TLS certificates to enable uh encrypted communications. So, let's start to think about some of the the major pain points when you're dealing with secrets. There are there

are three major challenges that you might typically face when uh when thinking about how to how to deal with your secrets. Challenge number one is git repos and container images. They're really not great places for your secrets. Baking your secrets directly into either source code or config files or container images might seem convenient at first, but it's it's a high-risk shortcut. hard coding them makes them readily accessible to anybody who manages to access your container registry or your repo. And you might think that making your repo or your registry private makes it safe, but the data shows that private repos are nine times more likely to contain secrets than public repos. And it's a false sense of um false sense of

security. um keeping them keeping them making them private doesn't necessarily keep them from falling into the wrong hands. Private repos are leaked online with alarming regularity and all it takes is one malicious insider or miscon misconfiguration or just a small oversight and suddenly your your private repo or registry is is exposed and made public. And the problem with with them being hardcoded is that they can be very difficult to rotate. Once you once you discover that a secret might have been leaked and that that secret is baked directly into your application code or your image, how do you then go on to rotate it your updated code you you need to update your code um which will once

you've made the commit to update it will have to then go through potentially a lengthy CI/CD process. Your container image would need to be rebuilt. your application recompiled and then you need to redeploy your entire application uh taking the old containers down bringing the new containers up one by one. So this all creates a lag when applying those critical rotations and it can greatly increase the uh the window of exposure.

Slack and email are also bad places for secrets. Secrets aren't just a coding problem or a config problem. They can be a people and a process problem as well. They uh they don't just leak through code or through config files. They might become exposed after having been shared over insecure channels like Slack or emails or WhatsApp or even a magical companywide shared word document. These channels they they lack proper access controls. Uh they don't have any encryption quite often meaning that the secrets can easily propagate beyond their intended recipients. Um attackers know this and they're increasingly combing through leaked chat logs, archives, email mailboxes for anything to exploit. Just don't make it easy for them. Don't leave your valid secrets

scattered throughout your your chat history. Challenge number three is developer workstations are also bad places for secrets. As your application grows into more and more cloud services and your workloads become more and more distributed, it becomes really difficult to keep track of everything and where where all your secrets are. You start to encounter then the the dreaded secret sprawl. Secrets end up scattered around unttracked in a disorganized manner. They start appearing in all sorts of places, your end files, your CI/CD configs, and your developer workstations, and maybe even post-it notes. Um, as a result, the secrets get duplicated and mixed up between multiple environments, uh, multiple cloud platforms and team members. It becomes nearly impossible

then to know which secret is the source of truth. uh your secrets live far longer than they should because nobody knows whether they're still in use, who has the latest version of the secret. This all comes this all this comes hand inhand with the sprawl is a lack of auditing. You really just find yourself blindly hoping for the best. Unauthorized access can go unnoticed and your secrets easily start slipping through the cracks. So if we take a quick look at some real world examples of secrets mismanagement and how that's led to to to real world breaches. The following is just a few examples that were caused by exposed application secrets. And the intention of showing you this is not to shame any

one particular company or person or engineering department. uh I just want to highlight to you that this can affect even large mature organizations with very wellrespected engineering teams. In 2022, Toyota accidentally committed a GitHub access token to a public repo which gave access to to Toyota's backend app for customer telematics data. And this led to the data of over 290,000 customers being exposed for a full five years before anybody noticed it. And this obviously drew a large amount of public criticism and it led to much additional uh regulatory scrutiny. In 2022, an attacker found hardcoded credentials after gaining access to one of Uber's private GitHub repos. These credentials led to a companywide breach unlocking Terraform dashboards,

AWS instances, and many other internal systems. It led to the exposed personal information of 57 million customers and 600,000 drivers. In 2024, an attacker gained access to Disney's internal Slack channels. Within Disney Slack workspaces, they found code snippets, API keys, and other credentials along with a 1.1 terabytes worth treasure treasure trove of IP propri and proprietary project data.

In 2021, 26 different Indian government departments were all found to have accidentally hardcoded their end files containing environment variables and secrets into their repos. These exposed access to personally identifiable information, police records, and citizen databases. And the list really goes on and on. We don't have time to cover the exhaustive list, but there's a lot of examples that you can pick from. So let's start thinking then about how we might try to improve things. The fundamental problem isn't that secrets exist. It's just how they're stored, how they're shared, and how they're accessed. We need to get them out of your Slack, get them out of your GitHub and out of your Docker Hub and and your config

files. and instead we're going to move them to a secure centralized vault which should help you to keep track of them all all in one nice location rather than being scattered across your various sprawling environments. With this you create a a single source of truth for all credentials. You eliminate the secret sprawl and the risk for the and the need for risky sharing pro practices. And with your secrets manager, then you're able to dynamically inject your secrets from yeah from your centralized location directly into your into your application at runtime. So they never end up hardcoded in your containers or your repos. Having this runtime injection then means that your application can automatically and dynamically retrieve the keys as

soon as they're changed all without rebuilding or redeploying and going through that whole len lengthy potentially CI/CD process. So you so you've got your secrets in your secrets manager and some of the benefits there from then on that it provides to you is you'll have encryption at rest of your secrets and in transit which will help you potentially satisfy compliance needs if that applies to you. You'll get instant response. So as soon as you need to re revoke or rotate one of your secrets, you can do that quite simply and easily. you just go into your secrets manager dashboard and do it from there or with an API call. Um, and you're not scattering around

trying to find where that secret is to to rotate it. You get versioning and speedy roll back. So, if somebody does make a a mistake whilst rotating one of your secrets, you can quickly, you know, quickly undo it, get back to your previous state, protecting you against mishaps and misconfigurations. You get granular access control. So you'll have an integration with your identity provider and you'll get a clear accountability chain. And finally, you'll get a comprehensive audit trail. So you will get to log every access, every modification, every rotation. You'll know exactly which service or which person is touching your secrets at any given time. So all this should transform your secrets management strategy from blindly

hoping for the best to something you can measure and something you can log and something you can act on. So choosing a secrets manager when it comes to your choices there are myriad options out there. We really don't have time to compare them all today. So I just like to focus on what I consider your three best options. First is AWS Secrets Manager or whatever your cloud provider's built-in native tool may be if not AWS. Your second would be Hashi Court Vault. It's a very established player in the game. And thirdly, you've got in physical the new kid on the block and it's got a lot of promise. So which of these three should you

choose for your specific use case? If you're a single cloud architecture and you don't care about vendor lock in, you're all in on AWS, your choice is probably quite simple. You're just going to choose the native tool. It integrates very nicely with the rest of the ecosystem. Um, in the yeah, in the case of AWS, that's either sequence manager or the parameter store, both of which give you pretty much all of the benefits that we just previously touched on. The challenge then comes to if you want to avoid vendor lock in. If you don't want to be relying on any one particular cloud provider or you're using some combination of multicloud, hybrid or onrem then either vault or in physical would

probably make a fine choice for you. Both of which give you a lot of flexibility. They can offer you a self-hosted version or a managed SAS option.

You might choose Vault if you want a very mature battle tested solution with a bewildering array of integrations for even the most obscure use cases. But be warned, it does come with a steep learning curve and it may be unnecessarily heavy weight for simpler use cases. It is noteworthy as well that it has caught some criticism surrounding their license changed. They used to be open source and they changed to source available. Um so if an open source license is something that's um important to you, it's something that's important to me. So uh in physical may just hit the sweet spot for you. Uh it's very easy to get off the ground. You can get it started

in minutes. It's got great documentation. It's got a much lower cognitive load and maintenance overhead than than Vault does. Uh, it's very developer friendly. It's got an excellent CLI and SDK tooling. This is a quick glance at the invisible user interface to give you a click a quick idea of what the dashboard looks like. It's clean and it's intuitive. So, you got your secrets there in the middle. And your secrets can be separated out and organized by project or by environment. You just quickly switch between using the toggle. You got your menus for all of the access controls, cloud integrations, audit logs, all the things that we've previously mentioned that you should be looking for in your in your manager.

So you've gotten all of your secrets now into your secrets manager. And from then how do we get them into your application? So with in physical there are three main patterns for fetching your secrets. The first method is with a sidecar agent. So this is an additional sidecar or separate container that runs runs alongside your existing containers. they may share a volume mount and your sidecore container will will fetch the secret um from your secrets manager and and and write it there to your shared volume. With this approach, it's very quick and easy to get started. You don't need to make any changes to your application code or your Docker files. Um it's all just done with uh with yeah

with config. Method two would be with an entry point CLI tool. Uh your secrets are fetched using a CLI tool in the container's entry point script. So you're starting to mess around there with the um the Docker file or or the the scripts within the the Docker image itself. The benefit of doing it in this way is it works with virtually any language or runtime or a CI/CD tool. Method three is you may use an application SDK. This directly injects the secrets into the process of the application. This gives you maximum control and flexibility over exactly when and how the secrets are accessed and the maximum privacy as well. So even if your container is breached, you still

need to the secret is is access is restricted to the actual application process itself. How do you know that application is genuine and is allowed to access SQ? Got chicken and egg problem. >> If you're using um if your application's on a cloud provider, your cloud providers, it links you with your identity access management system with the cloud provider. You have secretless secrets authentication between your application and your secrets manager. You're >> just moving the potential threat somewhere else. Yes. How are we up to? Um, here's a really simplified, really quick, um, quick look at some of the commands. Uh, I'd love to do a full live demo for you, but um, I decided it might be a little bit too

time consuming and maybe not offer as much value as I would have liked. So, here's a quick glance at a couple of of the commands you would run to get off the ground. You simply apt install in physical the CLI tool and oops and and then fetch your secret. In this in this example here we're um we're we're just defining an environment variable. So you might also write it to a to a file instead. >> And if I get that API key, what can I do with it? >> The the API key. So if I get hold of it, what can I do with it? I'm not I'm not sure I follow that which the API key to authenticate between the

connection to your secret to then go and get your secrets >> potentially. Yes. >> Right. >> So where you store that API key secure >> the thing I was talking about with it linking in with your um cloud provider's identity management system there there's an integration between the uh secrets manager and the cloud provider. So the you're not necessarily authenticating with an API key between your application and the secrets manager. It can be done in a secretless manner. So in closing, we've seen that secrets they're everywhere and the question isn't if you're managing secrets, it's just how well. If you want to do one thing after today, start scanning your codebase for secrets using a secret

scanner. You don't need to solve everything all at once. Just look for one secret. You can move to a safer place. And every secret you protect is one less breach waiting to happen. Thank you.