BSidesSF 2019 - Containers: Your Ally in Improving Security (Connor Gilbert • Connor Gorman)

then we'll cover some of the ways containers challenge existing security programs but on the flip side the opportunity they pose to enable finer grained more automated security enforcement and finally we'll spend a majority of the time showing you how you can use runtime monitoring to come up with more secure configurations so at a very high level you can think of containers as the next step and the evolution from physical machines to virtual machines infrastructure operators when a pack workloads closer and closer save some money achieve higher utilization and developers but especially the people who are paying developers want to move faster and deliver more value in containers especially when you pair them with an orchestration tool like kubernetes hope

you do that in some sense really containers are just a repackaging of existing things so Linux processes control groups kernel namespaces layered file systems cryptographic hashes virtual networking and some other stuff but containers really took off when it got easier to use all those things together instead of having to roll your own developers could package up applications with images with all their dependencies no more using ansible apt-get install this version of Python or that version of some pip package globally and they could share those images and run them anywhere and show their configurations in version control the same way that they'd version code and the containers would for the most part run the same way whether they were

on a laptop and a dev cluster or even in production with a defined clean state or clean slate of the initial file system and the defined configuration and soon after that Orchestrator started to appear other tools that helped you use containers more effectively they helped you handle things like scheduling fault tolerant storage all the other stuff that creeps up once you get out of the lab or off the laptop and into making real money with your apps and maybe everybody at Google knew this for ten years like most things but for the rest of us it became really clear that having containers and crude ideas and api's like this having standard interfaces to all the key parts of your application

was really powerful and that power could also change the game for security we walk through all the data found in a single speck for a single simplified application so this is a criminai just deployment spec with some of the details collapsed and I was worried it would be too small but that's like 6 feet per letter alright so let's start right at the top of the spec we see some operational metadata that's this stuff so like who's running this thing and what is it really we've got a little pointer that's exciting so labels and annotations are really important concept in communities the labels help you establish what the thing is and the annotations add things about

the app and you can see that they can be used to tag important info on in this case we're using things like who's responsible for this application and how to get in touch with them a lot of the people we work with try to add that as their first set of annotations so that when either security or ops or whoever is debugging a problem can immediately say I know who to ask so they don't have to figure it out themselves next you'll see the first bit of content that says what the software is and this is an image reference so it says a registry in this case docker Daioh or docker hub and then a name and

a version and importantly you can take this reference and you reference it to a cryptographic digest of all the stuff that's in the in the containers initial configuration in the file system continuing on we see an audit sort of the things that the container will have when it starts up so this is collapsed it will see it a little in a little more detail in the next slide but it's what does it have access to what secrets what config files what storage and then actually something that wasn't always next to the app code before how is it exposed so that's in the well sorry you had to sit up really high but you can see it's exposed on a port and in this

case it's on a load balancer or you can't see it so I'm told you so we can zoom in on the description of that containers object that I collapsed in the previous slide and here we see the same sort of image reference but there's a bunch more so at the top we see you know what will run so when you start this container what exactly will be the command that starts and how will you begin running the application next are there any guardrails so a lot of times people deploy containers and they don't really bother to inform the scheduler sort of the workload profile or 70 limits the limits can be really helpful to prevent runaway apps either

because of a mistake or because someone's trying to do basically a denial of service sort of you know use up your resources and start going through your nodes especially memory limits can cause containers to go super haywire it's the noisiest of neighbors next you can see some privileges so here we're dropping a capability we can see if the filesystem is read-only or writable what will be in the environment sometimes secret stuff don't do that and then some things like secrets block devices config maps things like that and beyond that because yeah mole doesn't get too well into into a slide you might have a bunch more stuff to I might have network policies that say who can talk to what

more storage options saying what kind of story do you want SSDs you know standard disks that kind of thing configuration Maps so the thing is used to have like an engine X config file you can put into a config map and you know mountain health conditions so that career IDs can do a better job of telling you know whether your app is up and healthy and running and you can even have things like custom resources if you get really deep into it and there's more stuff that's the theme of this there's always more but we're giving you a couple things now that's a lot of information and a lot of it might have been in you know a VM image or baked

into some system D file or something before but it's really powerful to see it all laid out in one file that's version control that's human readable if you can read yamo it's not not everyone but all of that's you know gonna be saved somewhere and critically you can see it later after you have deployed the same spec that you use to ask for it to run you can catch later and see you know once it's running so just a caveat before you keep going there's a ton to learn and a ton to say about containers and communities and they're really powerful tools so we can't cover everything about them in 30 minutes there are some really innovative

projects springing up around the community and each part of the infrastructure too and a lot of folks are mixing up the stack now here's my son deciding his stack I think one of the key concerns for a container project is you have to have a good logo and then the sticker table becomes like a toy a toy island for a two-year-old I think he picked cryo and maybe do you Iser and I think he's running a studio and prod like everyone says they are so anyway they're obviously a ton of amazing research projects and developing projects going on the spaces have all been quickly and the community has really been than innovating especially as the interface

is solidified around key components but we're just going to take one view of the scene here and concentrate on configs that you can adjust while you're deploying each of your apps there's a lot more you can do when you're setting up a cluster to secure the infrastructure you're like running on top of when you're building images you can take a lot of measures to slim them down and make them more secure more concentrated and when you're deploying them and obviously once you're running them you should do some monitoring too there's a lot of great content out there from koukin docker con a bunch of other talks you know blogs that you can find you should read everything I didn't I

didn't want to curate a list and leave anybody out and in fact if you just stay in the room ten minutes after our talk my aunt Anne will give another talk on a different angle of container security okay so what are we gonna show we're gonna walk through a couple of the security controls you can apply to kubernetes workloads we'll show you how you can figure out the right settings to apply and then then you know merge that configuration and lock down the app zooming out a bit if you're a person who cares about security and your organization uses containers we hope you'll see some of the power and not just the new challenges or risks that

they bring even if it seems like everybody else is just concentrating and getting stuff out the door I get it to prod they would want to change up the stack because it's fun we think that there are ways you can use containers you can use this change to the benefit of the security posture as well and perhaps you can bring techniques like the ones the the counter will show back to your dev and ops teammates and maybe lead a conversation around how to bake security and earlier in the lifecycle now could be a security in general it's a complicated topic but we'll spend a few minutes here so containers often bring greater scale with them but across

to a number of dimensions you might not think of so the way apps are built is changing and many apps are getting bigger much bigger and much more complex before you might have had one monolith running and you know some some ami or something with relatively isolated network maybe and now maybe your your your developers and operations folks are trying to leverage cloud services or break up pieces into micro services so that they can replicate scale up in parts independently that sort of thing you might have a lot more moving pieces and that means there are more things to understand before you can really crock what's going on in the app but the scale comes in the human realm too they're

more cooks in the kitchen no you have dev second ops maybe they're the same people but I haven't found too many and they're trying or they're not trying to stay out of each other's way so a successful security hardening and defection effort really has to involve all those folks and on top of that the landscape has also shifted we think in a way that can leave security as an afterthought so if dev and ops are on the cluster and security to outside that team they don't really have any natural hooks into the process but it's not all bad news containers and kubernetes and especially the standard interfaces that I talked about before they could really help you

do security better they don't necessarily allow you to do anything that wasn't possible before you could do many of the same things in a different stack and lots of people tried but the we do think they make things more feasible you would have had a lot harder time getting the same level of visibility and more importantly a harder time iterating on and testing those config changes without a stack that say enforces the declarative configurations that we saw on the screen before it enables the cost of those changes to happen be tested without rebuilding a full you know costly the end image or something like that and as in a lot of realms security free Tanners is sort of

a config problem there may be places where we need a new tack or new features but there already a ton of ways to configure grub rails limit security barriers and often the problem is really operationalizing the controls we already have if that sounds wrong just a reminder that anytime somebody makes a secure system somebody else do means the door so this is a satellite news truck brand-new Tech in 1984 literally next in 84 not like them not that would be it's at the Newseum the DC it's a great museum that's closing so you should go before it closes and if you look at the display case there's sort of this little like uplink thing an antenna warning

radiation don't stand in front of me and if you look around the side of the display case you can see oh great like there's some sort of pin lock or whatever wouldn't want anybody getting on TV without being supposed to but if you look a little closer okay pin code 1 2 3 4 and so somebody's JIRA for secure password distribution didn't make it into the sprint we can all end the ties and I can to go as an engineer 2 or 4 months ago now I just make in mad so another promising aspect of this you know stack is that security relevant configs live right alongside the app so if you're developing a new lockdown

profile you can test it in the same tool and process it's used to test new patches or new bug fixes or features you can even iterate on that on your laptop if you're using something like mini Kubb or dr. for Mac they make it really easy to just run a kubernetes cluster on your laptop I think Conor probably has one on this laptop right now even though we've got a presentation and stuff and if you change the app later for dev changes the app and it needs a new privilege just on your lockdown config then they can change the config right there but that's visible and it's committed in the same PR and you can see the same change going

on and then you know test the change throughout the the normal you know deployment cycle if you're using micro services design patterns or building out some other units of services in your app each atomic unit of the app also is getting smaller and that means that you can walk it down further understand it better it also means that those services are usually talking on the network which means that regardless of how the app works inside overall this if you're mixing different stacks you can use the network as a sort of equalizing factor to see what's going on between them it could be a lot easier to spot problems and deviations to if you're running apps

that have it like a single process or just a couple processes that start up as soon as the app starts up versus the sort of you know running a VM that's got some you know crontab going and running all sorts of things every so often much easier to understand what the default baseline ought to be and so really as people from ops and security communities really we're excited to see their broader use of kubernetes and we do think that the design patterns it comes with and the power that it brings leaves a lot of untapped value for security other people can tap those configs you move off the defaults and start locking down the application in a

way that we haven't seen before so we're gonna talk about three different types of controls that you can use to lock down your containers during deployment as I mentioned you can can test these changes so you've got a you know very simple deployment on this side of the room just get my app running and on the other side we can see the three things that we're going to talk about so you can opt into a read-only root filesystem hopefully spelled out instead of encode or whatever you can see we're dropping capabilities from Linux and also applying a network policy that allows some traffic to this application so let's see it for each of these we're

gonna gather some runtime data and then use it to guide us in applying the security optimizations without that kind of upfront collection it can be a little hard a little bit of a frustrating guessing check thing it's sort of just the config and then run it through CIC if it breaks figure out why some end-to-end test is failing and it's because you like disabled one little writable path the app expected was going to be there and that said you might be surprised what behaviors that's not getting even if you thought you know what your app did sometimes libraries will use like temp and put a lock file there or something and and you wouldn't have realized it okay so I'm gonna hand

it over to other Connor to talk through these capabilities and how his demo goes sure cool so thanks Connor yeah so basically what I put together is a listener that runs on every host it's a kubernetes daemon set it runs with privilege because I need to collect some really low-level kernel data and then it basically pipes it to a server which is we'll call it an analyzer for this sake but it's really straightforward way of just kind of querying the data and so as Gilbert mentioned before we're gonna talk about read-only file system the next capabilities and network policies in the con construct of single Apache struts attack so if I slide over and now is the fun

part where I just hope my demo works so let's see by paying the API great so the first thing that we'll do is we'll go ahead and run struts all right so I have a GK cluster five nodes these things pop it up great so running struts and this is the part where I'm really crossing my fingers maybe maybe not let me kill this pod I know I have a timing constraint my code somewhere and I have no selector so I can't select anything anyways I'll slide back to the slide real quick this is what you should see so what I've done basically is we're listening to the filesystem of the container and we're listening to all the paths that are

being written to or created and what it does is it pipes off to the central server and I can figure something called a potential filesystem root right so this container currently is not running read-only but you can see that basically the only path that's being written to is user local tomcat right so all apache tomcat servers right initial configuration data there and actually if you make it read-only it completely fails in your app is broken right so instead of just turning this read-only on what you want to do is say okay what path do I know based on what's actually running right and infer that from there so I'll slide back the demo here so

here's my struts app let's say I say okay fine let's not make it read-only and let me can run this exploit so this is using the the very famous 2017 straat CVE - you know download a crypto mine here from from github get it started using the command injection bug and you can see now it's running the original Java process but also the crypto minor right so at this point someone is stealing your electricity and your CPU this is one of the least malicious ways that you could probably exploit this bug but it's an exploit nonetheless right so what we're gonna do here is we're gonna actually replace this with a read-only file system but we've added the volume

line to the docker file which tells docker that hey this user local Tomcat is going to be read/write and the rest of the file system will be read-only so now we've applied this read-only file system and we're going to try to exploit again and this kind of shows really nicely and the purpose of this demo because I can scroll down a little bit but tar actually says oh hey by the way this is the read-only file system right so this is just one layer of attack that you know that we've kind of mitigated simply because we had a read-only file system a lot of attackers will use the normal writable paths as a place to drop

files right so slash temp is probably the most famous example a lot of Metasploit modules use slash temp because everyone else expects it to be writable as well but in this case if it's you know read-only then not even mitigated a payload drop for example the next thing I'm going to talk about pretty briefly our Linux capabilities right and so if I go back here right I can see that what I've done is I've used BPF probe written by Brendan Gregg who's actually talked about at besides two years ago and what it does is for the Linux capabilities that are being used and Linux capabilities are specific code pass in the kernel that required privilege and most people have never

heard of them and fewer people have used them and so I think it's a really powerful tool of like really scoping down the attack service of your container so as you can see this struts app didn't actually use any capabilities but tar actually did a bunch of file modifications and exercises those links capabilities and so what I'll do here is I will draw I will modify the struts app again this time I'll make the filesystem read/write but I will drop the Linux capabilities and then we can see how that kind of man that's when I try to run the same exploit so I'll rerun this exploit this time we should be able to download the

files and I'll scroll up again but actually what actually ended up failing in this case is setting the ownership right so we've basically stopped the CH own command from even running but Tomcat doesn't need that in the first place right so this is like layer 2 of basically dropping the capabilities of your container so if someone was able to run a script they would need you know specific capabilities potentially to try to exploit your system now you've dropped those capabilities and kind of mitigated that portion of the attack right so two layers is always better than no layers right and so the final piece is how do I stop the W get in the first place

right like how do I make sure that this struts app is never supposed to connect to the outside world instead things are supposed to talk to it right and so looking at the API I piping the TCP accepts and connects and we can actually see the W get right it's currently reaching out to the outside world downloading the crypto minor and so what I'll do is I'll apply a network policy I will call deny egress network policy and what that does is basically say hey this API server type you know connection type egress we're gonna head it and just block all activity right so we basically blacklisted egress from this specific specific container and in this case what

we'll do is we'll just apply this

so now I've applied the network policy should be active I'll rerun the exploit that to help it all sorry and we actually get like a little bit of a different air in this case right so our our code says okay well W get it's still running and the reason it's still running is because it's exercising its timeout trying to connect to the server right before it was pretty instantaneous in this case what W gets actually getting is a gateway timeout kubernetes is blocking that external connection and instead we are not having the crypto minor run because we couldn't even go to the internet and download it right so this is kind of three major layers where

you can say okay let's block network activity let's drop the privileges that are running inside the container and then let's make the container filesystem read-only and now you have kind of three angles of which you're blocking potential attacks and attacker needs to be way more sophisticated when they want to exploit your container because they have to really try to figure out okay this path is really like read or write okay this capabilities are what I have right and they need to like you can't just run a script and scan the Internet and try to exploit this because it just won't be that simple all right so we're getting the the signal I'm just gonna skip through the backup slides here

we've shown today some of the ways you can use runtime monitoring to figure out a better deploy time configuration which would limit the containers privilege level and mitigate certain classes of attacks especially those that people apply a sort of internet scale you'll block at least the first the first wave and attackers were lazy just like everyone else we really believe that containerized workloads and the technologies that power them really can help make the apps more secure than what came before in large part because we can see the security configs at runtime and iterate on them through the same sort of development process so that let us deliver apfel you fast the default configs are pretty decent

but you can really up your game by just opting into a couple more improvements and with that we're done so you can use the website to ask questions and I think we've got a mic out of this theater is huge so you can do him a solid by speaking up you know hey cool talk um so actually I've used Linux capabilities and I thought normally you only set them on one binary but it looked like in your config you were setting it for like a whole container so how does that work yeah sure so in the container config of communities and also you can either add or drop capabilities and so you can really define it for the container as a

whole right and then I think the processes that run within the container all inherit that probably starting with hid one and then going down exactly so it's a really neat way and but it's kind of hard to use right and so the idea is let developers develop and and push their code and then say hey by the way here's how you can make your configuration better because I used to work in infrastructure and I've seen you know things run with privilege that don't need privilege but they just don't understand the exact capability it needs or the exact privilege it needs Thanks so are you reading this the next capabilities has a root user or a normal

like just go back to the slide like this like to get this output how are we running it so these you are running from the host right not from the container itself right so actually what's happening is so we have a container that's running on every host and it's actually listening to a function called called cap capable which is where basically processes will ask hey can I run this this do I have this capability and then we're collecting that correlating with the pids and then correlating the pids with the containers right so in container lands a little bit of an extra work to figure out hey which container is running rich process and then we can take that

process and say which capability it's trying to use but the monitoring does require privilege on the host it's that what you're asking yeah so yeah ideally you do this once or every so often or something and then then turn it off sorry we're at time now so there any other questions I when I guess your be around yeah yeah okay and also you can find us on the Internet so thank you very much happy happy besides us up