BSidesATL 2018 - Containing the Cloud (Wes Widner)

my name is Wes Wagner I'm data janitor a heavy sleeper my daughter asks why I put that in there it's because I work on the cloud team with CrowdStrike and every now and then we're on Pedro Duty rotation and right now I'm on permanent Pedro Pedro beauty rotation so that kind of feeds into what I'm going to talk about here in this weird way that I don't want to actually have my page so I'm going to build systems that don't get broken so a little bit of context first when it comes to containerization security and all of this why am I here why is the name of the talk containing the cloud well this just happened this past week

and so since it's public I get to throw it out there we recently were FedRAMP ready not authorized FedRAMP ready there is a how many of you familiar with the FedRAMP or any of that so amazon has a special cloud that they built out for the government that is called gut clack and we call yeah everything that's an overloaded term in order to get into it you have to pass all kinds of audits checks pin tests everything this was our last pin test that just completed to three weeks ago so now we are ready to sell to government organizations for organizations in general now I want to get into that platform and I've been on

the team that's been working on this for about the year so that's sort of the context here what that means is that we're using kubernetes in FedRAMP or for our gum cloud installation and we were FedRAMP authorized a big question going into it was how would we get to do that when we're one of the first not be first but one of the first companies to use containerization in a highly secured environment so we're also using it for our security application so we're helping to secure other people and we're doing it for highly secure for security conscious organizations they're very opinionated now they want things to be run so that kind of ups the states and also didn't

put into here that Google is one of our investors so we also have pressure from that side too so all around lots of screen that's the context from wondering Brown coming from on container securitization so being on this team for about a year last year I did like a month or so just reading through white papers of what people have written about containerization and security and stuff like that and part of the the cool thing about this conference the standing on the shoulders of giants containerization is nothing but shoulders of giants over the years nothing new about containerization what's new about it is how to lose together some other points here and these are part of the slides of

those just put it together like literally five minutes ago polling shows the primary reason for people not going into or not using containers is security they worry about how that how that works now that's become less and less over the last couple of years but it's still something that you probably have running around on your mind in fact there's a lot of organizations that just flat-out know so I'm coming at it from this high secure conscious all this to let you know that it is possible to come to implement containerization securely and properly I also want to point out that I'm coming at this question from a security or from a systems engineering perspective I'm a software engineer I

happen to work for a security company so whenever I'm in a conference like this I mean a lot of you guys that work in Sox and that's great that's cool I'm the guy that runs the stuff behind the scenes that help support you so we um we just crossed the billion hundred billion event mark per day to put that in context our CIO ourselves was saying that that was about going par with like Facebook and a few other large companies so we're we're ingesting an awful lot of data that doesn't leave a lot of room for air that means we need to be secure but we also need to just get the job done so

what our containers first of all how many of you are using containers today anybody okay doctor containers anybody using something other than docker containers in any application nobody so containerization is actually an overloaded term it could mean like a whole bunch of things iOS Android lowest in Windows they all have the concept of containers that are baked in that's where the entire industry has been moving for quite a while now containers I would argue are all about application deployment now in order to put that in the context I put that slide in the wrong place in order to put that into context so let's give a brief history of software development life cycle SDLC the simplest deployment

is a self-contained binary okay you know what I mean when I say that just an executable everything LinkedIn we throw it over the wall the problem with that is we kind of like dependencies software engineers we're a lazy bunch we like to use what other people have already put together so we end up using libraries that introduces that introduces another layer of complexity now I have to know what library I'm depending on after that word version of that librarian is and I have to bundle that library up in order to ship it with my application consistent deployments then this is where this really comes in consistent deployments meaning that I have to find that same like bitwise same deployment

every time okay I'm moving into aiming at a certain thing here system purely peculiarities calls deployments to be complex now there's been several security related incidents in software deployment recently NPM repo someone yanked their package and it broke like a whole buttload of other packages more recently though Python had one of their repos compromised somebody was publishing like nefarious packages that kind of sounded like something else that's another issue so software deployment is complex one author said that software development is one of the most complex things that a person can do today I think that they weren't entirely exaggerating so this is the appropriate place for the xkc so the xkcd comic here in a nutshell is everything is a micro

service if you throw out most of the features and that's basically where containerization has been for the last year or two and this is not entirely I would argue this isn't necessarily a bad thing if you want to go play with something like elasticsearch or cassandra Kafka or something like that horribly complex software by itself I don't really care to go track down all the configuration if somebody wraps it up in a nice container that I can just docker run and get it running right there that's cool but that's not going to carry us over into production that's where the critique here is so our deployment options first we started off with single binaries that we did package

management systems we still use those they still get a lot of use still have their niche then we move to virtual machines and virtual machines are great that's where Amazon built what we now know as the cloud we're not just Amazon but many other companies the problem though is that VMs are heavy and slow that is you have to literally boot an entire system go through the entire boot sequence by contrast containers are deployments of virtual virtual machines they boot up in milliseconds not seconds and they give us so this is what they give software engineers they give us the familiarity of running in a VM that's all of our own and they with the speed

of process execution so as a software engineer I like to talk you PI the entire system just ask my kids when we watch them even I like to explore the whole space I like to do that when I'm soft when I'm engineering software - that is this network interface is mine all mine nothing else and that's a good pattern to to go to because that's where a lot of software started breaking and is where we have mixing them - so since we're at a new folks that conference now that we've gotten the deployment part out of way here's the attack service on a bare metal machine you've got all your apps running on the rest OS and the

hardware is accessible right there that would be the least secure VMs on the other side have apps running on their own OS abstraction on site on top of a hypervisor that actually provides the abstraction to the hardware now containers don't have the hypervisor containers are running in the host OS on the hardware and you're probably wondering well why are we supposed to get more security with containerization no not at all and I want to point this out the more secure is running in a VM actually the most secure is running in your own machine its own space but because of something because we need our applications as tightly as possible to get the most out of all of our

Rackspace all of our compute power we need to figure out a happy medium that's what containers are the containers so put it in another perspective there are 12 VM escapes I searched back all the way to 2007 there are 12 VM escapes these are hypervisor attacks ok so ok 12 p.m. escapes and I want to point out something here to you most of these are escapes that are off of like random crap like virtual hoppy or virtual floppy disk controller or just things like a fusion SPG a driver should be running that anyway you don't need that on your videos so it actually shrinks if you're doing your VM deployment properly by contrast the primary attack surface for containers is

the kernel itself ok almost 2000 kernel exploits back to 2010 ok and that's because the kernel is a really complex just it's really complex ok that's what we're up against we removed with the hypervisor layer that means we went from 12 VM escapes now we have almost 2000 ok that's because we don't have this intermediate layer so that means so a container is really just a slaw djegal separation of a set of processes files and other resources then the kernel just says we're going to set this aside ok we're going to treat an application like a logical unit so what we've what we've got here is a vulnerability for all of our outdated dependencies and attack

vectors we need to track and stand both the base image as well as all the application libraries that that base image has in it then now that means that security begins at container creation one of the first things that we did when we set up our entire with moving into a containerized space was we have a system for putting together the doctor files for our developers because fundamentally I don't trust software engineers I'm one but so we start off with the way the container is built up each layer our images are built up by layers that's the copy-on-write file system and each layer has an operation that's performed on the layer below so it starts with the from of a base image and

that's where all security begins as making sure that you're using a secure base image and then each command stays with that build so one thing that we do is we don't put secrets into our build scripts we have to trust the build machine we have to also figure out how to tag our images so that we know that they came from the build machine docker in particular has a nice cryptographic hash that goes with the content the content of every image we also use labels for auditing purposes when somebody says well who push this image then we could go back and say here's where they push this image on this system at this time and all this other

stuff and then we use what's called the Builder pattern when it comes to container creation the Builder pattern means that we use one image that has all of the helper libraries all the div versions of libraries to build the binary and then we have a different image that we deploy that has all the statically linked stuff in it so we have a fact image in intermediary that has all the stuff that we need all of our cool developer tools and then we strip all that stuff out when we go to put it in production this gives us a unique opportunity security is actually usually bolted on after the fact and software development containers give us

the ability to put security as part of the workflow in building an application and crown spec we have the philosophy that security shouldn't get in the way of what you're doing it should fit alongside it so this gives especially application security engineers an opportunity to come along and audit your docker scripts your base images your libraries and then then provide a helpful way to like steer that into a better way so the continuous deployment this quote I love when healthcare gov the they went out and they asked industry experts and one of them came back and said the first suggestion that you would have was President Obama to have the team deploy to production every

day and that has that was a software chemistry thing in the software industry or the software engineering that's a really good like deep like make your software less complex makes it make it easier to set up but as a security of operation it has another implication too in that the less snow flaky your stuff is the though easier it will be to just tear it down put something else in its place so continuous security is what I'm arguing that we're going to move to here the way to move forward is with DevOps tooling there's all kinds of great stuff and like I said in the beginning power of standardized security till they can fit in each part of the software

development lifecycle when it comes to container security I do want to point out one thing though that containerization is not so containerization is not less complex it's even more we get every bit of complexity from from the lower levels on up so it ends up being this wave of complexity so think of every piece of this deployment process as an opportunity for you to inject good security practices yeah there is all the software supply chain issues are apply to containers the one thing though that you can look at whenever your managers ask you why are we putting in all of this added complexity at the end of the day it will sign off when it comes to

velocity in deployments we have this really great system set up it uses spinnaker which is Netflix's on development tools or deployment tool for containers and all that and it is really complex you should never use that for a side project but as a engineering as a industry thing it's really awesome because it's got all these folks and checks and stuff built in and in particular security checks so here's what I mean by all the complexities of the software build lifecycle you've got the developer to build things and commits it to github or to get a repository somewhere somebody could take control the repository the image is built by something usually a developer on their laptop and they push it up to a

registry which they shouldn't do and then that registry is what deploys that into running containers in an environment every every layer of that and cannons should have eyes on it from a security perspective so we can automate the good news is we can automate a lot of the security tooling another tie into the theme of this today of standing on the shoulders of giants Red Hat has done amazing things with container security not particularly docker necessarily but they put a lot of thought into you wouldn't believe how many white papers and lectures out there rooting the container no longer Maps to root on the host just want to throw that out there but it's still problematic

file ownership gets mangled between the host and the guest read or use read only if you have to the tight coupling between host and a guest is a micro service fail treat that as a code snap or a security smell shouldn't do that if at all possible get those off containers need to provide a minimal containers provide minimal protection I'm assistant actually don't so just treat your containers your guests containers as possible and that'll give you a good path forward real quick I'm running out of time here that's why I'm trying to talk a little bit faster it's up to us to use additional protection measures it's really easy to shoot yourself in the foot with

containers you could just pull something straight off github or docker hub and just start using that and it's very easy to let yourself be lazy and put that straight into production don't do that use app armor SELinux to further harden your images these are security profiles that you can you can add to the the container running engine usually docker and further secure your your containers as they run or what was released this week Google's chief Iser system which I encourage you to take a look at it's actually a shim that sits between your application and the kernel itself and provides additional security trace your applications kernel calls to discover what capabilities it really needs and

then shrink-wrap those capabilities so that your app doesn't have the ability to do things like access of floppiness which it never means without the possibility of doing so to wrap up real quick don't worry in your application is route don't write to disk in production audit your deployments in configuration is code I can't stress this enough the pattern for containers is to apply environment variables to container as it runs in runtime okay there's all kinds of patterns to get away either either to get away from that for to make that better there's a sidecar system that LinkedIn uses called Envoy which is really really good I encourage you to use that pattern or if nothing else

track and version all the configuration stuff around your application because I promise you the webpage tonight and somebody changed some random variables somewhere and the first slide kind of heavy sleep alright because I had to run through the slides real quick I'm sorry but the good news is I've captured a lot of this awful lot of this in my container security awesome and I encourage you to go and look at it and especially star it I treat stars as love so show me a lot of love thank you guys [Applause]