BSidesATL 2020 - Connect: Flex Seal your CI/CD Pipeline

it's your entire IT staff outsourced twenty-somethings is your cloud environment architected by a thought leaders fevered dream have you swallowed heaping gallons of the DevOps kool-aid only to find out that you're one msconfig away from losing everything Oh Sean Marshall here too and I invite you to flex seal your CIC D pipeline if you're interested in diving deeper into this topic my entire slide deck is on tiny si slash flex seal there's entire slide deck is also on download slide channel and slack so feel free to go either way and now transition to the about Me slide so three things I do i code I teach I hack so I'm a developer and cybersecurity consultant for secure

or ghen called secure ideas we do penetration testing red teaming and we focus heavily on web applications and cloud infrastructure assessments we also do network tests and things like that as well and we also do continue continuous monitoring and scanning using AWS there's there is a full professional bio of me in the link in the slide deck below but I don't know about you I just want to jump into the topic so here's the ground that we're going to cover first I'll define what a sea ICD pipeline is and then we'll focus on three strategies for funneling security throughout the whole process if you do not have any idea what I'm talking about what Flexsteel just Google that out real

quick just watch it after the talk you're in very good tree so the three strategies we'll be focusing on are based on flex seal family a product flex glue will flex glue different components together using cloud formation flex tape our packages and dependencies while we're building in the pipeline and we'll also flex seal the build containers that we'll be using and have a demo demo straining all of this so ACI CD pipeline is a system that allows multiple versions of the source code to be integrated built and rated for deployment in a perfect world we get from repo to production in seconds in the real world we either end up with something that's either painfully slow

or something that forces out way too much information at once the focus of this talk is to bake in as much security as we can without sacrificing efficiency or availability this is the kind of stuff that we need to integrate and existing systems so that tagline it works under it even works underwater should apply to all of these strategies that we're going to talk about there's a lot of tooling available in this space so like Travis Jenkins get lab and a myriad of other players and this talk will be centering around code pipeline why because this is a tool I'm most familiar with and therefore it is perfect but now it's a decent tool because it's relatively easy

to setup and it allows you to automate build and deployment processes this can be as simple as a two-step employment for ns3 bucket or it can be as convolute it can be as convoluted as you would like and when you use code pipeline you will get temporary access to a build container with the AWS CLI installed bash and now going internet access the other tools that we will discuss will fit neatly in this but first we'll think we'll need an example now imagine for a second you're building an online portal and it's going to be so you'll deploy the client-side application to an s3 bucket all the requests will be handled by lambda functions grouped under arrest-- API to

find an API gateway and user information and user group will be distort in DynamoDB now each one of those lambdas will need to access different api's and other AWS resources at different levels of authorization depending on the user the authorization level of these are making the request does that sound convoluted to you trust me you have no idea but these are the types of applications people are building nowadays so it's perfectly normal to have a cloud native application that speaks to dozens of surfaces now the question is how do we tie all this stuff together how do we do so securely we could we could have engineers creating things like through the console manually but that process is

error-prone and there are way too many opportunities for mistakes here and there so we need to tie things together cohesively and the first fundamentals of that is understanding I am in AWS so here's the crash course I am or identity access management has to do with users or entities acting on user's behalf there are hours of content to go into each one of the topics that we discuss here but we'll stick to a general overview for now users and groups are self-explanatory right we'll be focusing most of our definitions on we'll be focusing most of our definitions on I am roles and I am policies so let's start with an example I'm a user in my AWS

environment and I have two roles as a developer I have access to source code and whether it's in github or eight or AWS code commits the Amazon based solution as a consultant and as part of that consulting world I also have permission to spend up an ec2 ec2 instance to crack passwords so that makes sense the roles are difference s of one or more policies each one of these as JSON documents specifying what a user can and when a user cannot do so if you now not only our users this not just users in groups that are aw that are can be defined with AWS rules and policies AWS services have their own rules and their own policies

so for example in the serverless web app that we discussed before the pipeline is has its own role and part of that role it needs permission to access github and s3 and the permissions needed to build the source code or do n deploy it in the cloud if there's a cloud formation stack if there's a confirmation template embedded more on that later and of course s3 has its own special set of policies called bucket policies but that's out of scope for this talk messing up on your IM rules and policies can make your AWS environment and secure and you can also mess up your i''m roles and give services access that they don't need so in the capital one breach for

example then the attacker uses C surf to get on the ec2 instance she was able to query the permissions that the ec2 instance had by accessing the manatee that accessing the metadata so she's like oh the server has full s3 access well why don't I just dump the data for all 300 buckets and see what pops up there so if Capital One and exercise the principle of lease privilege making sure that the policies in line the policies and role attached to the services only have access to the things that they need then the breach wouldn't have nearly as much damage so secure use about so secure use of AWS involves mastering I am roles and

I am policies here's a reality here's a reality bit of reality most of your code is written by someone else everything as a loose collection of parts stuck together we're talking API is micro services libraries open source and legacy clone and cloud native applications are no different not only do they have all of those things they also have AWS services cobbled together as well and so this is where you need to flex slew things together these is where cloud formation templates come in so you can think of CloudFormation template as a blueprint for provisioning cloud services and infrastructure think of it like a UML diagram except this document actually helps you create working software so you

can push a button and deploy an entire stack for an application based on a single JSON file you can provision most AWS services including I am roles and policies using information templates so if you template your entire text stack you can replicate modify and version control it and by doing that you can get something a bit more elegant so if we're going to be gluing things together anyway let's flex glue it together now here's a bit of homework for everyone that has node in their stack you can use the package manager or to audit your dependencies for vulnerabilities you can do this with NPM on it or you are not it if you're a hipster trash like me and

there's several tools that could do the job like sneeze and github stupenda bot but this is free you don't have to install anything extra and there's no con figs that you have to mess with often you'll find that these simple tests produce nothing you all your vulgar built all your dependencies are up to date and they don't have any active vulnerabilities but sometimes you get a laundry list of vulnerabilities ranging from low to critical so a lot this tooling is available in c-sharp Java Go and many other languages and a lot of tooling is of is open source now if you want to fix some of these problems we can just tack that on the

end now never not all patches will be easy some will have breaking changes but a lot of patches are just three words on the command line this can be done in an afternoon on the staging amp or it can be done by one person over a course of a week if there are breaking changes rather than spending the time and money to get your red team to point neces point the SAST and DAST tools that may not even work or burp suite and other tools at the web apps you can eliminate and detect a lot of the look hanging fruit in the pipeline itself so you can fix leaks in your dependency by patching your depending with flex with this sort

of flexed ap sort of way now we go on to containers with code pipeline AWS gives you provision container with deployment controls now many of you are already using containers in your environment and tech stack but it's just doctor containers that you manage yourself or multiple docker containers in a kubernetes cluster that in building and deployment of your services and products the world of container security and container hardening is vast and growing but here are some tips that I can offer first is do not run service anymore services that are needed okay not every single container in your orchestration cluster needs access to orchestration tools needs access to needs access to a whole suite of UNIX tools if I am in a

tap if and if I'm an attacker on an engagement and I Papa can i Papa container one way to make my life really difficult is to eliminate things like curl and we eliminate things like curl and make it difficult for me to reach out to the Internet and download my tooling into the container to try to reverse shell and pivot throughout so you don't need it's not a p.m. you don't need to pack everything in the kitchen sink into your containers when you're building and running them second you need to understand that docker access is route access the docker daemon runs as route there is an experimental version of docker currently being played tested right now that involves rootless access

those link to that is available also in the side deck where you can find those resources to try to play around with that in your environment but as it currently stands docker in production right now if you give the user access to docker by putting them in the docker group you're giving them root access I can there's also another link in the side deck to show a simple privilege escalation in about 10 seconds it's not that difficult so you need to understand that not you need to understand that systems need access to docker and not necessarily users then you need to understand also avoid docker bind mounts like the plague now find mounts are mind mounts are insecure because they

open up the file system of the host to the container and it's fine for development local and play testing purposes but when you're shipping out containers in a production style of environment you need to use volume mounts in step and then finally you need to understand that there's an excellent meme going around about docker and a corona virus but that isolation between apps is not absolute it's not a VM it's a compromise between running virtual machine which could take minutes and in terms of DevOps and cloud infrastructure that centuries and then just running it on the hardware itself it's compromised between the two so the dot so any container shares are the kernel with its

host so if you keep the host systems kernel patch you can avoid the content you can avoid attackers who pop the containers to avoid those kernel level vulnerabilities and in order to escape the container and so if you follow these steps you're well on your way to you're well on your way and improving the security of the containers and your environment now we jump into the demo so here we've got a brief demo with a portal that we've discussed in the beginning of the talk as we go through we notice that oh the build failed what's let's see what's going on and that's investigating the logs and when we dive into them we find that all of

our other services work we've deployed the cloud formation stack the unit tests work the API support correctly but we notice that there is a moderate level vulnerability and we'll see that at the end of this log right here now I don't know about you but my boss will not allow me to ship code with cross-site scripting in it now if you're working in a system that has a high availability requirement maybe you would instead trigger something in cloud watch to alert any incidence response or patch management or the developers themselves to try to quickly figure out a patch to address the moderate level of vulnerabilities that wouldn't necessarily trigger an ultimate shutdown and not shipping and not shipping the D

product into production what you do in your environment depends on your tolerance for risks and your processes for vulnerability management and so you may ask alright so give me the tool that automates all this work I'm sorry I don't have it I've shown you tools tactics and strategies to help builders and security teams to be more put me more effective there is no handyman in a can none of these tools replaces the people that must use them so whether you're a builder or an InfoSec you're a plumber you're either laying down pipes checking for leaks or fixing them and by strengthening the connections between parts analyzing the building materials and preventing data analyzing your building materials and preventing data

leaks you can help keep the flow of information going throughout your organization I thank you for your time and I thank you for your attention and now I told you I'd run through its lot