Flex Seal Your CI/CD Pipeline

Is your entire IT staff outsourced 20-somethings? Is your cloud environment architected by a thought leader's fever dream? Have you swallowed heaping gallons of the DevOps Kool-Aid only to find out that you're one misconfig away from losing everything? O'Shawn Marshall here to convince you to Flex Seal your CI CD pipeline. If you're interested in diving deeper into this topic, my slide deck is hosted at tiniesteye.com. Feel free to ask questions during the talk. I'll be monitoring the Slack channel for Q&A.

So who am I and what do I do? I code, I teach, I hack. I'm a full-time developer and DevOps engineer. The things I talk about in this presentation are from my experience of trying to implement security, best practices, and production facing systems. I also have experience explaining how to do things like this through videos, blogs, and live classes. I'm not only a developer and an instructor, but I'm also a full-time offensive security consultant as well. I've seen organizations who architect the cloud native applications well, and I've broken into web apps that aren't designed so well. So,

Here's the ground that we're going to cover. First, I'll define what a CI CD pipeline is so we're all on the same page and then we'll focus on four strategies to funnel security throughout the entire process. All of these strategies are inspired by the Flex Seal family of products and the beautiful things about the products and the commercials for them are the demos. Each one of these topics will have a brief demo illustrating the topic. Well first we'll start with some foundational knowledge on IAM, then we'll flex glue different components together using CloudFormation, flex tape broken dependencies using Node, the package manager, and we'll flex seal the build containers that we'll be using.

So, a CI-CD pipeline is a system that allows multiple versions of the source code to be integrated, built, and ready for deployment. In a perfect world, we get from repo to production in seconds. In reality, we either end up with something that's painfully slow or something that's efficient but leaks credentials and data all over the place. The focus of this talk is to bake in as much security as we can in the building, testing and deployment phases pipeline without sacrificing efficiency or availability. Now there is a lot of tooling available in this space like Travis, Jenkins, GitLab and a myriad of other players. And this talk will be centering around CodePipeline. Why? Because it's a tool I'm most familiar with, therefore

it is perfect. But no, it's a decent tool because it's relatively easy to set up. CodePipeline is a service that allows you to automate build and deployment processes in AWS. Now this can be as simple as a two-step deployment of a static website to an S3 bucket, or it could be as convoluted as you would like. Of course, GCP and Azure's has their own versions of this service. But first, we need a more concrete example. So now imagine for a second, you're building an online portal, but you decide it's going to be serverless. All the requests will be handled by Lambda functions grouped under REST API defined in API Gateway with user information and user group stored in DynamoDB. Each of those Lambdas will need to

access different APIs of other AWS services at different levels of authorization depending on the user. Now, does that sound convoluted? Trust me, You have no idea. But these are the type of applications people are building nowadays. It's normal to create cloud-aative apps that speak to dozens of services. But not only do you have multiple resources for multiple different services, you also need to manage the identity and access management within the entire tech stack as well. So in order to do that, let's do a quick crash course on identity and access management in AWS. So I am for short has to do with users or entities acting on users behalf. Now there are hours of content that can go in each

element discussed here, but we'll stick to an overview for now. Users and groups are self-explanatory. So we'll define I am roles and I am policies instead. An I am policy is simply a JSON document that specifies what a user can and cannot access. An I am role is just a set of one or more policies. So let's start with an example. I am a consultant and a developer in my company's AWS and So I'm simply a user with two roles as a developer. I can access company source code and various resources that only developers can access and as a consultant I can create a server for cracking passwords for an engagement. Not all entities are an AWS are users or groups of users

non humans like Amazon resources also have roles as well. So let's say you're using CodePipeline to deploy that serverless web app we discussed earlier. CodePipeline is something with a role. Now the role for that pipeline would have to have a policy to give it access to pull code from GitHub and deploy the built product into S3. Now of course, S3 has its own set of policies called bucket policies and we'll cover the difference between IAM and bucket policies better in the demo and differences there. But suffice to say, messing up your IAM roles and policies make your AWS environment insecure. You can also mess up your IAM roles and give services access that they don't

necessarily need. In the Capital One breach, for example, you have a public-facing web application that was compromised. When the attacker got onto the underlying EC2 instance, she was able to query the permission that that EC2 instance had. So this server ended up having access to over 300 buckets, each having confidential data. So if Capital One exercised the principle of least privilege and only gave that EC2 instance access to specific buckets and files and the breach wouldn't have nearly as much damage. So secure use of AWS involves mastering IAM roles and IAM policies. The tools I'm going to talk to you about in these demos are freely available Everyone using AWS and I mean free as in free not free as in free tier and then

it's $300,000 bill at the end of the month so you can use it however much you like and it actually can help you craft better policies to enable access between different services so

We're going to start first with the AWS Policy Simulator.

So, the IAM Policy Simulator is a useful tool in that you can simulate accessing, doing actions against AWS resources as any user, group, or role. this simulation that we're going to do we're just going to go through a trivial example it's DevOps gone wild and you've got and you've got a bad S3 rule that's just full that just gives full S3 access so

we go to S3 we select all the actions And we run the simulation. And therefore, and there we go. Green means go. Everything is allowed, right? But this, you have to think about it. Green means go for anyone who has access to this role. That includes in that example before, that EC2 instance that was compromised. that if that EC2 instance is compromised, it has full reign to do whatever you want in S3.

So anyone who has access to this role can not only change notifications, can rewrite and modify files in different S3 buckets. A lot of people use S3 for logging, for security logs and information. A lot of people in S3 use S3 to store SSH keys, for example. So this so having this much access can actually which is akin to admin straight up modes can.

Can and will affect the security of your AWS environment so.

When we think about S3, we're only thinking about a simple set of use cases. As a user, I would like to download files. As a user, I would like to upload files, generate a list of files by name, and that sort of thing. So in a sample of a good S3 Demo Policy, we have those sorts of resources We're getting a list of files. We're uploading. We're allowed to download files, and we're allowed to upload files. Awesome. But we're only specifying that on a single bucket, because usually you don't need cross-bucket access. Now, when we rerun the simulation, we get access denied for everything, and that's actually a good thing. because note that the simulation resource here

is star so if we want to change it up and you want to specify okay all we're looking at all objects within the S3 bucket meaning all files then what we would do is we would apply that we would take the ARN that we had on the left and

and then we do just slash star

and then

we can also specify all the buckets there just by typing in the ARN there and those for the bucket level permissions that we're doing now when we rerun the simulation got for the specific actions that we needed downloading uploading and listing and listing the file contents of a particular bucket all of those calls are allowed we run the simulation we can see that all of those calls are allowed but let's say we go through development of our web app and we find out okay we need to all this web app also needs to be able to delete files because that wasn't specified in the secure policy all that all you need to do for that

is just simply add an action

we're going to add the action delete object to the secure state policy and Once we apply and then rerun and when we rerun the simulation you see that it's denied, we rerun it and now we see that it is allowed. So

the IAMPolicy simulator is a tool that allows you to just run these actions against various AWS services to see what you can, how you can craft a policy that fits your use case. Now let's say you're not a mad lad and you do not want to write JSON live in front of a studio audience. So we jump into the AWS policy generator. And so in this case we're doing an IAM policy over the service S3 and specifying the same actions that we did before. Get object to download files. put object to upload files delete object to delete files of course and the list bucket the list bucket operation which is allows you to refer to the bucket by name and refer to

the bucket by name and view its contents so delete get

bucket and put

and why why these particular actions versus any other actions you there's no going around having to know the actions for different I am policies that's fundamental to how to get AWS components to work resources within AWS and to be more specific to communicate with one another so knowing which action does what that there's no shortcut there's no real shortcut for that other than referring to the aws documentation that last part that i'm just building that's just filling in is the arn amazon resource name and in that case all i'm doing is just saying hey do it for this bucket and for all the files within the bucket access by both in separate ARNs and now we add the statement and we

generate the policy and ta-da we have the entire policy JSON document available now let's see and let's see if I cover brief tangent into S3 bucket policies so

that's fine so we are create we opened up the we opened up in our discussion what's the difference between a I am policies and s3 bucket policies with I am policies you're more egress rules how do you get one resource or component to speak with another resource and component S3 bucket policies are more ingress focus from the outside world What are the limitations of what can communicate into this bucket and one things that is really difficult to do in IAM policies on S3 versus S3 bucket policies is forcing encrypted encryption on all uploads so

I'm going to add a deny statement and star. This is a policy that's saying no matter who you are, if you're accessing this bucket, here's accessing this bucket, specifically if you're uploading files, here are some conditions that we're going to apply to the operation. And since this is specifically just a put object operation, we can just do an ARN of a slash star instead of also having to specify the bucket. And so, hey, we are denying all uploads to this bucket

under this specific condition. And so the condition is...

if the encryption header isn't correct so the encryption header is S3XAMZServeSideEncryption if it is string not equals and that is AES256

if the SERP then what we're saying is whatever requests are made to this bucket if the server-side header is not equal to AES-256, it's wrong, we do not want the upload. So we're going to add that condition. And then just add that as a statement. But with this statement alone, it's not enough to force encryption on uploads. We also need to add another deny statement. So deny, and that's for everybody who's connecting to this, and the put object operation

and of course the ARN with all resources slash star under this condition if the encryption header if null if the encryption header AMZ service side encryption is not even present so

if the encryption editor is not even present then we do not then don't want to bother with the upload so we're adding that statement and from there we can generate a policy this one is This is the policy recommended by DAWS to enforce encryption on all uploads into a particular bucket. And there we've used the IAM policy generator to create S3 bucket policies and more stronger and secure IAM policies.

to be switching over to Parliament. Now we hop over to VM just to do a brief demo on Parliament. So we do a quick LS and we see a single JSON file get with put. Let's cat that real quick and we see that it is just an IAM policy just placed in to the

structure of this VM so we know the get object option action now the second one the put bucket policy action has some serious security implications

but you wouldn't necessarily see that in the simulator alone if you didn't know what that action was so this is where the parliament tool comes in so

we're just going to use Parliament and then we're just going to feed it this file and I'm in the way of things let me fix that real quick

there we go so this this particular policy that we have above has a leave some avenues privilege escalation one of the major ones is that hey if you have the ability to put an object the bucket policy into s3 bucket you can change the permissions therein and from there you can escalate you can escalate privileges you can make the bucket policy do whatever you want By changing the bucket policy you can make the S3 bucket open itself up to you in however way you imagine. So

that's one of the benefits and one of the powers of using tools like Parliament. Switching back.

And now we'll go back to the slide deck.

So, everything is a loose collection of parts stuck together, whether it's APIs, microservices, libraries, open source, and legacy code. Now, cloud-native applications are no different. They are glued together pieces of a dozen or so cloud services. So now the question is, is how do we tie all this stuff together? Now, we could have engineers creating things like this console, online console manually, maybe script around some of the link, the API's released for Python and JavaScript. But this process is error prone. There are way too many opportunities for mistakes here and there. And this is where CloudFormation comes in. A CloudFormation template is a blueprint for provisioning cloud services. Now think of it as a UML diagram, except The main

difference is that this document actually helps you create working software. You can push a button and deploy the entire tech stack for an application based on a single JSON file. And you can provision most AWS services, including IAM roles and policies in CloudFormation templates. So if you template out your entire tech stack, you can replicate, modify, and version control it. So instead of cobbling something together, you can make something way more elegant. So that leads us to the next point. So what's the value here for security teams? Now, if you have infrastructure as code into the code base, you can analyze the code for vulnerabilities before the code is deployed. This is a demo of a tool called CFNNAT. It looks for patterns and

CloudFormation templates that indicate that it will deploy insecure infrastructure. So as you're building your CloudFormation templates,

you may be running this tool time and time again. And in this case, while we were in active development of our portal, we looked at the CloudFormation template that we're generating, We see some linter informational type errors and warnings, but we also see that some of the

rules that we were creating were way too permissive. So there we go. If you're using CloudFormation to build your stuff, you can make the whole gluing together process more efficient. So if you're going to be gluing things together, flex glue it together.

Now this is a little bit of homework for anyone who has node in their stack. So you can use the package manager to audit your dependencies for vulnerabilities. And you can do this with the NPM audit or yarn audit if you're a hipster like me. There are other tools that do this job like sneak and the github's dependent bot, but this is free and you don't have to install anything extra. Often you will find these tests produce nothing, which is good news. Sometimes you will get a laundry list of low to high vulnerabilities. And so that creates the process. Easy question. How do you patch of these vulnerabilities how you work through them and here you go that's the solution

tacking fix on the end of it will automatically apply patches for various dependencies now some upgrades do have breaking changes so involve some code change in your code base but a lot of patches are just three words in the command line This can be done in an afternoon in an app in active development or one person a week for a large enterprise app if they are in fact breaking changes. But rather than spending the time and money to get your red team to point Nessus, Birch Suite and other tools at your web apps, you can eliminate the low hanging fruit in the pipeline while you're building the web application. Now there's similar tooling in this. C sharp Java go

and many other languages and a lot of that tooling is open source big names include OS Dependency check OSS index and retire.js so you can fix leaks in your pipeline by patching your dependencies

and Here's where we put all that together. So here's a brief demo of the deployment process of the portal that we discussed in the beginning of the talk. Now as we go through, we notice that the build execution failed. So let's hop into the logs and see what's going on.

When we dive in and actually look at the investigation, we see that we deployed the CloudFormation stack properly. The unit tests work, the API was deployed correctly, but we notice that there is a moderate level vulnerability. Now, I don't know about you and your type of work environment, but my boss will not let me ship client-side code with moderate level vulnerabilities. But let's say that your web application has a high availability requirement and you can't stop a build or can't stop a release. You can adjust the settings of NPM audit or Yarn audit in your environment to only block high or critical vulnerabilities. Or you can set up a special alert triggered from this event that notifies

patch management or the developers to apply the patch and fix. Or you could use NPM audit fix and see if, oh, do these particular unit tests work? If so, then just deploy the patch automatically without any human intervention whatsoever. What you do in your environment depends on your tolerance for risks and your processes for vulnerability management.

So with CodePipeline, AWS gives you a provision container with basic deployment tools, but they also allow you to specify your own container. Now some of you are already using Docker or multiple containers in a Kubernetes cluster that aid in building and deployments of products and services. Now the vast world of container hardening is growing, but there are some simple tips that I can offer to you. First and foremost, just don't run more services than are needed. A VM isn't a container. You often need it for only a few simple things. You can make it difficult for an attacker to pivot or move laterally in your system just by limiting what utilities are installed in the machine from get-go. And also,

by limiting access to container deployment and orchestration tools, you can make it that much harder to pivot up the chain. So, you also need to understand that if you give a user Docker access, a system you are essentially giving them root access now there is no there is an experimental version of docker that uses rootless docker that will allow you to use the system without giving the user root but that is not necessarily as of the time we are talking right now it's not been made into the production-facing systems. And three, you need to make sure that Docker containers are not VMs. They share a kernel with the host. So patching the host systems kernel is very important to maintaining the integrity of your

containers. And if you follow these steps, you're well on your way to improving your container security. But to illustrate

that second point, we discussed before giving users root access is the same giving them docker access I should say it's the same as giving them root access here's this brief demo above me where we're just pretend that we've compromised a username quill we look at the groups and we see that quill is part of the docker group and so let's see from here if we just really quick

this command let's see and we have root

and so how is that what is the structure of that command so the command that was used breaks down as follows so we each at each level we just open up various elements of the host to the container whether it is the device's kernel functions, the networking stack, the namespace, then the root directory, then mounting the root directory as a volume mount, and then running chroot. The best way, and I do mean the best way, to prevent these sort of problems happening is simply to control who is allowed to run Docker on a given system.

Now, you may ask, where's the tool that automates all this work? I'm sorry, I don't have it. I've shown you tools, tactics, and strategies that help builders and security teams be more effective. But tools don't build tech. Teams build tech. There's no handyman in the can here. None of these tools replaces the people that must use them. So whether you're a builder or an infosec, you're a plumber. You're either laying down pipes, checking for leaks, or fixing them. So by strengthening the connection between parts, by analyzing your building materials, and by preventing data leaks, you keep the flow information going for your organization.

Thank you for your time and attention.