Do Containers Fully ‘Contain’ Security Issues? A Closer Look At Docker and Garden

to go we're gonna go on track two

all right welcome to the final session of the day at b-sides in track 2 we have a little just a brief announcement for those of you that are either speakers or paid for VIP tickets just want to remind you the VIP dinner is from 5:00 to 7:00 p.m. and what we were calling the VIP room and if you haven't fully explored the rabbit warren that is this building the VIP room when you came in this morning down the hallway where you were checking in to get your badges you just when you get to the end of that hallway instead of only left in to track one you turn right and that'll take you into the

kitchen area of the area where we're gonna have the VIP room and dinner tonight so for those of you that paid for that and our speakers please make sure if you plan on joining us please do we hope you're all there and our next speaker is Farshad and he's going to talk to us a bit about container security which I learned a lot last time I heard you speak on container security at the the OWASP meeting I think a couple years ago at Mozilla was was when I first started thinking more about docker security and things like that and and Farshad is an innovative technologist according to this piece of paper in front of me so

please give him all of your attention for the next hour and over to you for shot just before I started I'm curious to see how many of you have ever used containers in any way shape or form if you can put your hands up cool that's good there's a fair number of people because what I've done this presentation in different forms there it's sort of a mixed percentage of people but it looks like there's a fair majority of you that have had a taste of containers so the talk today obviously for the next hour is about container security again and we just want to look at whether containers actually live up to their names and

contain the security issues or not so what I'm going to talk about I will just go through a number of things we'll talk about what what is first of all and let's start with what is competing in the cloud what is cloud computing mean and then once we've you know sort of have a good idea of where all levels set on that let's talk about what are containers and how do they fit in that model and then what are the supporting cast so all the tools and frameworks that go with it because it's often times I find that there's a lot of confusion and how people throw these names around like kubernetes and docker and PCF and this

and that and they call it all containers right but they're not all containers some of them are supporting the supporting cast some of them are the actual container engines so we're going to clarify that before we move forward and then we'll look at the differences between containerization and virtualization and and then we'll look at the two more common ones that I'm familiar with it I know a lot of people in industry use which are which are docker and cloud foundry or pivotal foundries garden and then we'll look at some secure we'll go over some security concerns to see what what people should be concerned about and when deploying containers and and also what are some of

the no involvement abilities in the last couple years that these containers have been in use and then finally we'll look at safe container practice and we'll wrap it up with some points in conclusion so my personal journey with containers started a couple years ago as the company that I worked for started deploying these things and they decided that it was a good idea to jump in and we want to go into the world of DevOps and you know and I'm sure that's probably a good number of you have have become accustomed to using containers through the through the through DevOps and so we organization decided that we're gonna go and do this and we're gonna get you know pivotal as cloud

foundry or PCF as a platform and and you know so you know I was assigned to figure it out essentially and my background is I mean I'm currently the principal security architect for the leading financial institution but I've got a wide very very background in terms of where it comes to IT I've done a fair number of years in software design architecture development as well as the fair number of years spent in system admin and all that other fun stuff that people would do so this is this was really interesting for me and and you know I sort of have become that I guess the security SME when it comes to this at work just by the virtue that I'm

really interested in it so let's talk about cloud computing you can see here there's a cloudy Vancouver right we're looking at it from the North Shore this is taken from West Coover it's looking at downtown there's Lionsgate that's very cloudy right so cloud can mean different things to people so cloud can be a great thing right so you just put things out there you don't worry about it but I could also be kind of like Vancouver right so big gloomy and mysterious and it's sometimes it could be dark and you have no idea what's gonna happen next so this picture is a good representation of cloud computing and what I can relate to with respect to my life in Vancouver and

here's a quote from NIST in terms of the definition of cloud computing so they define cloud computing as the model for enabling ubiquitous convenient on-demand network access to a shared pool of configurable computing resources for example network servers storage applications and services that can be rapidly provisioned and released with minimal management effort for service provider interaction right so this is their definition how you accomplish this could be a number of different ways right so that's kind of what I wanted to talk about take a minute and talk about because you can in order to do that you need you need a workload isolation right so you have a bunch of workloads and that needs to you know meet the NIST

definition so how you can provide that isolation there a number of different ways that you can do that so you could either accomplish that goal through physical separation right so you could literally accomplish that goal but have physically separate machines where the workload is distributed and that's your separation boundary or you can go into virtualization or what's commonly referred to as virtualization but also refer to as hardware virtualization that's what you guys are all familiar with like VMware virtual box that kind of stuff we're all a little bit so in this case what we're doing is the hypervisor is your separation boundary and you can have vulnerabilities and in these cases related to CPU or hypervisor

issues that could have impact here and and this type of separation is typically used to provide infrastructure as a service or I as as they call it right obviously if you do physical separation it arrests us pretty much most of the issues with hypervisor to me you don't have hypervisors and you also address things like covert channels and defects and hardware protection mechanisms that are going to touch on in a bit right so physical if if you're really really concerned you would go with physical separation but most of us sort of you know move on to the you know you want the benefits of continuation or virtualization and with containerization your your separation boundary is what

it's called also known as operating system-level virtualization and your separation boundary is while you share the kernel but you have like multiple copies of operating system with a single shared kernel and and it's used to deliver containers a service or past platform as a service so what are the difference between the two of them well with platform as a service you get a whole bunch of extra stuff you might get a database in cash and you know logging services and things like that whereas containers the service is just containers as a service delivered as infrastructure pieces without all the bells and whistles that it has would give you I pass has kind of had been

around for a while the Cloud Foundry was you know one of the companies that is well known for offering their their pass or the platform as a service and then this the the containers of service is becoming more and more popular which is somewhere in between infrastructures of service and platform as a service and then of course you have SAS right multi-user accounts that's still a separation boundary but it's not as strong as the other ones right as you go down the stack the separation boundaries become weaker I mean with with SAS your separation boundary is your user accounts your sharing everything else underneath it even the application is being shared right but I mean if you're

doing has at least you've got your own application you're sharing the platform and then with you know with I as you know you're basically just getting raw raw devices so as you're going down this way the the separation boundary changes so containerization that provides an isolation mechanism required for cloud computing so we can consider it as a type of cloud computing so what are containers right so we talked about cloud computing in the different models for separation but what about containers so containerization as we said also known as operating system-level virtualization it's where a kernel allows multiple isolated user space instances and this concept of namespaces or user spaces is really what this whole thing revolves around so each instance

is a container so it's also known as the software container or a virtualization engine or a jail so in case of the FreeBSD jails or chroot and it seems like a regular mechanism machine from the inside right so if you were sitting in that machine everything would look like a regular operating system you wouldn't be able to tell whether you're in container or not but of course you know you're sharing that kernel with all the other could instant container instances that are running on that host and and so it in caps so it allows you to do in capsule applications and also containers can act as the interface of the surrounding system and and there's kernel provided resource

management so it limits the impact of the containers activities on others right so in the old days we had to route and chroot has been around since 1979 I mean in my old days in IT I used it a fair bit back in the day we were like hey you want a deployed application and put in at route you know harden my sequel and harden everything else and throw in the truth you know jail have you and you know that's what you did but true didn't provide all the other namespace separations I mean all they did was it sort of separated your your file file system right so you got your own route file suit handle and and you

wouldn't be able to see the real route if you if you were within that true tour the environment but you know if you tried to look at the process IDs you would be able to see everyone's process ID and and other things so it wasn't as fancy as what containers do today so it isolated the file systems for processes and their children right so if any children processes were spawned off of that you know they'll also see that that they would be within that isolation and but it doesn't isolate the other resources that I mentioned so it doesn't isolate networking or process space or anything like that and the root account has full of you and access into all the

different turbulent environments right so you didn't have that separation so containers came along you know I think it was around like early 2000s and what containers introduced was they took advantage of the namespaces to separate the resources so I'm gonna go through that in a bit of a detail but essentially you know we now have processed the process namespace the mount space namespace network namespace UTS IPC and user name spaces which are arguably the most important ones in this journey of you know security with containers and then there's also C groups that are used to meter and limit resources and control access to device notes for example like dev slash dev and and perform crowd control

so namespaces I kind of wanted to go through this I put lots of stuff here lots of detail but essentially just what I what I went over at the top I put the usually used the user name spaces right that's the most important one and that was the one that wasn't actually implemented until recently so a lot of the implementations like when I did this presentation about a year and a half ago when I first started in this journey of of containers and then containers security docker didn't support user name spaces and neither did Cloud Foundry and I think it was about less than a year ago that docker implemented user name spaces and Cloud Foundry he's also had

it for about the last year but even then there have been proven vulnerabilities that have been have been able to break out of the user name spaces so nothing is for certain just because you have user name spaces that doesn't mean your problem was solved but what problem does the user name space solve well what it does is it allows you to map a particular user inside your container to a different user outside that container right so if you didn't have that if you didn't support user name spaces the root user in the container would be the same root user on the host system right so as you can see the problem with that if if

it's the same user than if you've got a vulnerability in a particular container and that service is running as root then it'll have the same permission on the host and then of and because there's a shared kernel it'll get the same permission into all the containers so that's a bad thing right we don't want you know the root user that in a vulnerable container to be able to impact all the other containers so we want this separation we want this mapping so that you knows for example if there's a root user in a particular container it maps to a non privileged account on the host so if it breaks out it's not able to do a lot of damage so

it is a good thing as desirable a given user ID for example root inside the container will map to a different ID on the host it's fairly new and I'm as I mentioned earlier there have been vulnerabilities at the start when this was introduced it's very large and complex to implement as well not sure if everyone's got it right and that's another thing with containers this this technology is relatively new so more research needs to get done more things into I get ironed out I mean the rule of thumb as I always say in security and it's it should be it should have stood the test of time right with anything with an algorithm with their

system you always wanted to be out there for long enough that people have had a chance to you know hack away at it and find the vulnerabilities right if something is fairly new and not a lot of allman abilities affarin I wouldn't be too comfortable using that that doesn't give me that warm and fuzzy feeling that it's like hey you know this has been around for a year and not much has been found on it I'd be more comfortable if this was around for three years and lots of vulnerabilities that have been found and and addressed as such so system calls and security logic should be namespace aware that's another problem and you should check the capability in in the

correct namespace right so you might incorrect checking may lead to there's a clone new user in clone FS would exploit eyeball met abilities and exploit there's still possible even with with the user name space enabled as I mentioned so what about the other namespaces are they important as well yeah they're definitely important I kind of put them in the order of importance you know with the user name spaces being at the top and of course the network one is quite important because it provides a separated network stack for each container right so that you know you have these they weren't different network handles but and it's fairly widely used there's a natural abuse case that explores typical flaws in this

namespace if you're interested in looking that up there's also the process ID namespace you want to keep those separate as well you don't want to do a PS and one of your containers and see the list of everybody's process IDs right you want to be able to when you do that inside the container you want it to be limited to the process IDs that are that are that are that are that belong to that particular container so the paid namespace allows each container to have a fully isolated process tree so with an init process that runs as paid one in that namespace and then as you spawn off other things then it'll go from there so

the paid will be different inside the container that on the host involved but vulnerabilities have shown that this label can leak as well right so again I keep saying there's none of these things or a hundred percent leaks out of these namespaces can can't happen so you just because we have these you can say well no I'm perfectly fine and as it's secure and everything else there's also the mount namespace which is used to separate the filesystem for each container so there's a pivot root system call is used in conjunction with that as well and then last but not least we have the two down below they're not really widely used there's the IPC namespace

which deals with system five IPC and POSIX mess Q's if that's your thing and then there's the UTS namespace which it's got system identifier x' that's used to provide container specific host names so that's a bit about the namespaces why they're important there's also a couple of other things that are important in this in this story when it comes to container security and those are cgroups capabilities Mac and set comm so what are see groups some of you may have heard of them so maybe some of you may not have heard of them see groups are also known as control groups what they do is they limit account for and they also help isolate usage of things like

CPU all your resources right memory disk i/o networking for a collection of processes and it can also be used along with IP tables for traffic shaping right so C groups are important we want that resource limitation because if we didn't have them then you know one of your containers could go wild and use up all the memory or all the CPUs so that's how you get put a handle on it and then there's the capabilities right what are what are these Linux capabilities Linux capabilities allow you and the breakdown of the route role into smaller pieces that can be granted to non-privileged processes right so if in the past we had a bundle of stuff that the route could

do but we don't like well it'd be really nice if I could take a little piece of that and then allow this non-privileged process to just do that one thing right so Linux capabilities allow you to do that you can break things down and then you can assign a slice of that into a non privileged process so that they can perform that otherwise otherwise privileged action and then so a set of capabilities are signed to process they carries forward to the child processes as well from there so whenever you assign something you're like hey this capability is like to this one if that process spawns a trial process it it it's granted downwards to other things

that are important mandatory access control or Mac basically these are the LSMs or Linux security modules hooks are provided by those things so a farmer is one of those essentially what this does is these LS Em's or things like a farmer or SELinux they help limit the actions of a program so it can't just run wild and free and both docker and Lexy have this enabled which is a good thing SC Linux is also an option I mean a farmer is preferred everyone is raving about a farmer SELinux is an option but it's not as well supported and documented so the preference is given to a partner and then last but not least seccomp which is used to filter

system calls right so we don't want to have again just anybody called whatever system calls that they want to call that can be dangerous we have two different modes there's the strict mode which allows only a very small set of system calls to be performed and you can't modify it so if you're in strict mode that's not customizable you can't modify anything but if you're in filter mode then the filters you can write custom filters using Berkeley packet filters or BPF if you have programs which allow you to do more finely grained policies enforcement and things like that lexy supports simple policies docker supports second PP FS there's a P trace vulnerability that bypasses this is well

right there's vulnerabilities for a lot of these things okay so we've talked about a few these capabilities and and and security mechanisms but what do we use containers for it's just real quick here I mean like most of you are using them and hopefully falls within one of these bullet points here so you know same as Harvard ization in terms of allocation of finite hardware resources that's one of the main uses obviously it helps you with separation of several applications for improved hardware independence and added resource management right so we can package it annapolis dependencies in a container it also it's fairly commonly used for hosting api's and micro services that's what I use them for mostly that's why we

started going down the container path because we took our big old monolithic applications and we started to slice them up into into micro services or api's and that kind of goes hand in hand we're like yeah I've sliced these things up I need to put them somewhere let's put them in these containers and then you know there's stateless and you can deploy them rapidly and there's all kinds of other wonderful things that come along with that with that application so that's a fairly important use for containers so you typically move to call them application containers if that's what you're doing with them there's also like you could categorize them into application containers and operating system containers depending on

the use case you know they can be used to automate deployment of applications distribution method for software to guarantee reproducibility you know DevOps comes in right so DevOps tools for testing and deploying code that can be containers can be used as building blocks or paths right if you were being in a platform that is the one of the key containers of a platform as-a-service it can be argued that you could probably do platform as a service of DMS but typically paas is done with containers in the comment and virtual hosting and then the last but not least what about security was security a driver for using containers that's the funny one right because as the technology is maturing people just

like yeah well that isolation should give you added security but it's kind of like where VLANs where like back in the day when VLANs first came out for those of you remember they were not VLANs were not designed to give you security right like they invented VLANs to kind of quiet down the chatter and create separation and whatnot but it wasn't a security based intention right security was like a secondary to that and then of course to go we were doing VLAN hopping and all kinds of other stuff but as technology matured both through better practice and matured technology now we're comfortable with using VLANs and we're like you know nobody will argue that if you want to you know you want to

separate security in your network you can go with VLANs and that's a fine option because we're fairly advanced in that journey we can now we have the tools and the processes to use them in a secure manner we're not quite there yet with containers right so if someone tells you that containers are a number we're gonna use them for security it's not you know I mean my are old sisa was completely against them he said you know containers I don't even consider them a security boundary like VM is my security boundary because containers are really new right but that was two years ago since then a lot has changed give it another two years they will be a driver

for security but I personally wouldn't say that today myself so here's some modern implementations you can see cloud foundry obviously docker Lexi opened opened desire for Linux containers there's modern implementations make containers easy to use right so you know we kind of started with chroot which was not very easy to use even docker didn't have a lot of orchestration and a lot of that other stuff when it first came out but as these things mature they're getting easier to use they come with ecosystems and tools that allow developers to learn them more quickly and orchestrate things and schedule and like I said all the toys that we're gonna look at in a minute and the sort

of the ecosystem that goes along with it but the two common ones there I've listed docker Engine and cloud foundries garden and some of these containers are designed to run multiple processes and services some are designed to only run a single service so the ones that run are designed to run a single service we would call them application containers like I mentioned earlier you deploy a single micro service into them and that's all that's supposed to do but you could also use that container to run multiple processes and use it as a virtual operating system so that would be the operating system container all right so the supporting cast so we talked about the main actors right

that's art could docker engine and garden and things like that but it is all this other stuff that supports them because without the supporting cast you know that the actor and the actress would be sitting there kind of useless so what what are what are the supporting cast here well one of the more important ones are the container schedulers right so the schedulers allow the allocate containers to allow for scale-out load balancing adding storage or network resources maintaining high availability and the recoverability right these are all really important stuff so if you Bert is to use docker engine by itself you you wouldn't be able to do all this stuff so you need something like swarm

right so swarm is gonna take a bunch of your docker engines so docker engine we're gonna look at the architecture in a minute but it's gonna take a bunch of those and create a whole group of them or whole swarm of them and manage them together like that kubernetes does the same thing you know orchestration there's Cloud Foundry Diego right so if you're using garden you'll be using Diego for orchestration but these are all sort of at the same level right they all orchestrate the container engines whether your container engine is docker engine or garden or something else and then there's you've got obviously to package things and stage them so you know openshift open shifts source to

image or SGI is one of the more popular ones you also have the bill packs that Cloud Foundry supports clarity is now supporting a few other mechanisms for packaging and staging as well and then you also have infrastructures of service orchestration right because you know sometimes it may not be enough to have your container orchestration you may be working on a platform where you might need that virtualization like if I'm working on my Mac I might need to virtualize and use of and then use that container that's compatible with my my platform and vice versa so typically for rolling out the virtual machines that you can have the containers then you do that orchestration using Red Hat ansible or

something like cloud foundries box right so if you're a cloud foundry shop like what we are you've got Bosh's managing the virtual machines right so you deploy a virtual machine you orchestrated the VMS and then once you've orchestrated the MS you have to put a bunch of containers in the VMS right and that's where Diego comes in so we've got two different levels of orchestration right Bosh's orchestrating your your infrastructure as a service you've got VMs and then Diego is orchestrating the containers and doing all that type of stuff and similarly with with ansible and swarm right you can think of it that way so there's different levels of orchestration that come into play and

then of course you know at least the other supporting cast platform as-a-service to open shift or Kyle foundry itself right the entire thing would be a platform as a service and and these tools evolved at the same time independently so that's a bit of a it presents some challenges because they all were kind of happening at the same time and people weren't talking to each other so there are some compatibility problems and you know as the again as the as the ecosystem matures you some of these guys will drop off the map some of them will stick around so we'll see what happens but but it is it is a pretty interesting time for this

technology and there's a lot going on at the moment so the next thing I wanted to talk about is containerization versus virtualization because you're all probably wondering but what's the big difference right like you know yeah I've heard containers are light and VMs are heavy and but what's the real difference right what what is going on there so containerization I mean container ization container containers share the same kernel right so that's that's why they're good and that's why they're bad right because because they share the same kernel it's really fast to boot them up right you don't have to load the whole kernel you'd have to wait five minutes for whatever kernel it is to

load up and and all that kind of stuff kernels are you ready to go all you're doing is basically loading the container but that's not much to load right you might have some custom libraries or things that go with it but pretty much the bulk of it is ready to go and that's why containers are light because you're sharing it you don't have to boot that load off that kernel it's already ready to go for you so it's a single OS kernel provides filesystem access application process controls and networking and you can't use containers with different operating systems right because obviously if it's a shared kernel I mean if you're running a whatever Linux kernel well you can't

have a container that's got like a Windows kernel right it's a shared kernel so that's that's sort of it I guess it's an upside and downside because it makes things faster but it doesn't give you that that capability to deploy multiple operating systems with different kernels right if all the operating systems are going to run on that or all the containers will have to have the same they have to share the same kernel have to be of the same flavor so they're faster lightweight a more portable and they scale pretty efficiently there's no hardware emulation at all right breast virtualization right you're doing hardware emulation the hypervisor is providing an emulation layer in that sense and then then containers you see

groups of namespaces so you really are creating that separation through those mechanisms and they don't provide the same level of isolation as as virtualization now on the other hand virtualization is more mature with an extensive ecosystem right like VMware has been around god probably for 16 or 17 years now and they allow for mixed kernels right because they're not sharing so each of the guest operating systems that is running on the hypervisor it's gonna have its own kernel it could be whatever it wants to be as long as the hypervisor support sat down beneath there's no problem there and the host emulates the hardware provided to the VM so it looks looks like it is running on separate Hardware

right so that's where the hardware emulation comes in and the hypervisor is your security boundary in case of the VM right and and that would thereby it's more secure then then containerization by that nature I mean the hypervisors have been known to have Ballman abilities they've been exploited and people are broken out and the hypervisor so don't get me wrong it's not again it's not foolproof but it's been the technology has been around for a long time and a lot of those issues have been ironed out and again through mature technology and processes we can kind of trust hypervisors whereas you know you've got this kernel that kernels got a lot of potential issues it's a fairly big large

attack surface and you know if anything happens it'll impact every container that's that's using that so here in in this in this picture you can see that here's my type 2 hypervisor right so I've got you know my bare metal server there's a host OS and there's the type 2 hypervisor and then I've got my different guest operating systems and each of the full separate kernel they've got their own binaries and libraries there's nothing really being shared whereas with containerization of course we got your server and host OS but then there's there's basically that host OS kernel shared amongst all the containers they may be even shared binaries or libraries right so as you can see these

four containers they're sharing all the same binary library so if there's a problem and if one of these binaries or libraries they don't impact all of these containers here and similarly these two are sharing as well and again there's a problem in the kernel all of these are impacted but if there's the problem in this kernel the other two are not impacted so that's just a visual representation so let's look at behind the scenes right let's look at a little bit of an architecture all deep dive into you know what what does docker look like behind the scenes versus what what garden looks like so docker accesses an external virtualization in many ways so before a version point nine lexi was the

default execution environment for docker so docker went to Lexi got the Linux kernel but as as after version 0.9 run C which is used to be called lib container has which was written and go it's open source it allows direct use of Linux virtualization facilities so that's what they're using now for docker the other ones are still supported in addition the abstracted virtualization interfaces are still supported using libvirt and and an assistant D and spawn but like the primary way for docker to access the Linux kernel is now through the the run C or also known as Lib container and with with docker also the actions are done to docker base images so you have a

base image and then the Union filesystem layers are created and documented to allow for recreation so something nice it's a nice model and the docker demon though it runs us route right so just keep that in mind here's the docker engine architecture right like we're not talking about swarm just just the basic docker engine you've got your docker daemon you've got the actual containers and you got the client the client is the command-line interface but those of you used it but really under the hood it's just using restful api so essentially the daemon is exposing a bunch of api's and then they've created that command-line interface to call those api functions so if you wanted to you could also create

your own CLI because it's all based on the API is that are exposed but at the at the heart of it what the docker demon does is that when you try to you know when you go docker run or you name it and you name a package if the package if the image is already here it will simply take it and it'll spin off a container instance from that image and you know it might add the few stuff to it if it's not there it'll go to the registry and it'll fetch that image and then it'll spin up a container right so the kinds of things you can do through the client you can you can do a build

you can do a poll you can do a run and so you know if we go docker run something it's not there it's gonna go good the image it'll you'll see oh yeah downloading image because I couldn't find it and then it it runs that container for you and then you can take that so what you saw there that's like one one docker engine now you can put a whole bunch of these so if each of these were one of those docker engines you can put them together and then you could have this swarm scenario where you've got the managers and the workers and then now you can have a whole cluster of these things and you

know deploy things depending on where you want some some load some some workload to go right so one of the one of the use cases that I'm looking at basically is if you have different security requirements and maybe you don't trust the the container separation boundary and you think VM is more appropriate so you could have these different clusters and then you can tag your your applications that you're deploying and then based on the tag it can get deployed in a different cluster so if you don't want if you have a bunch of you know applications that are risky and then something that's really sensitive you may want to deploy that in its own node right so each of these

things each of that each of the container and docker engines would be a node so then these are swarm nodes and then you you know you've got a cluster of them together so you may want to create a cluster for your very highly sensitive applications you may want to create a cluster for less sensitive applications and deploy them such by through the use of labels and automation through your bill pipeline so what about garden garden is kind of similar as well you know fairly architectural similar is you got the garden client they had the doctor client there's an API there's a demon you know they kind of follow a similar model the main difference here is that the the

back end is as interchangeable so instead of having it all sort of in one they've they've separated it out so that you can have the garden server and then various back ends that actually do the container management so that's your ward and she'll demon and then the actual containers that get deployed so this is kind of like what you saw in the previous diagram where the docker container itself was and this relationship to the docker daemon so a garden was is actually warden that was rewritten go so previously a cloud foundry there they're offering what's called warden and then you know they came and rewrote it and and go and also changed a bit of an architecture they

created that separation that plug ability and that and then they renamed it garden and so essentially it has a pluggable backends for different platforms and runtimes specifying a set of interfaces that must be implemented so today actually they even support Windows containers which is kind of interesting so for example container creation so they sort of interfaces are are things like container creation and deletion applying resource limitations doing a snapshot to allow for redeployment of without without downtime etc etc so there's the garden run C backend so that's the Linux specific implementation of the garden interface that's it uses the OCI which is the open container interface standard previously was garden Linux right so that's in the

diagram in the in the picture here this was garden Linux and now it's run C but again this this is just as it's interchangeable and you know you can create different ones for different platforms app armour is used for all on privileged containers and set comp whitelisting restricts access to a set of system calls right so we saw what what a farmer was earlier and you can see it here in play and and then there's the warden shell daemon which is the root process for managing containers and launching the apps and essentially it's stream' standard output and error back to the client right so that's that's sort of the garden architecture and then here I

don't know you can see it at all cuz probably really small but I wanted to put this diagram because it's a it kind of puts it all together cuz when you think of Cloud Foundry I know for me it was really confusing I'm like oh my god Diego Garden Bosch this this this how do they all fit together this is actually a pretty good diagram so you know you can find it on the internet and it you can zoom in but at a high level all of this you know he says Cloud Foundry this entire thing is Cloud Foundry right and then here in this box this is Diego and then within Diego you've guarded so this

is this part of it so would they call it the the Diego cells which is the same concept as that the doc are the swarm nodes and docker right each of the swarm nodes it contains the docker engine each of the Diego cells here contains the garden engine essentially so you can see within this Diego cell you've got the you know there's the the executor and then there's the actual garden Linux container and then so within the Diego cell you got the cells you got the brains so that's kind of like the the workers and the managers and swarm for those of you who may be familiar with that and then of course you've got this

is a pass right platform as a service so you've got things like your database your access control you've got your longer Gator you know you've got all kinds of other physical your routing and all this other stuff that come together to create what we you know what is the platform as-a-service right and down here you can see that it supports their traditional build packs this was like the original how a garden or a warden used to work but they now support docker images as well and they also support the windows containers so the cloud foundry is really expanding their horizons so yeah here's a side-by-side comparison so as you consider the features there's docker garden sort of garden docker in

terms of resource isolation and control garden has CPU shares memory swap network bandwidth disk disk size quota docker does CPU shares CPU sets memory memory swap block device bandwidth in terms of dynamic resource management it's it's supported but it's not used by garden and docker doesn't really support it in terms of image management garden supports only hole images that can be reused to create new ones but docker has that nice layering which allows for reusing the separate layers right because we got the base image and then you can keep layering on top of it which is which is actually pretty nice and then can you link can you do container linking garden doesn't support it docker

does how about exposing ports they're both the same multiple ports but container can be exposed as such what about security concerns I mean I think that's why you're all here because this is a security conference right so so far we've been kind of talking a little bit about security or the architecture of these technologies but in terms of security concise we talked about the the namespaces right so the user namespace support is really important it's supported in garden and docker but as I mentioned it has been shown to be people can can break out of it and it's not enabled by default in docker so if you're deploying docker and if you really concern you should enable this

right because that's one of the most important namespaces and without that processes running as root inside a container can have root access everywhere I so it's it's rather important that but it has to be configured property right mentioned I meant that remember I was mentioning earlier because if you don't configure a property you may not check the right namespace and then you you may you may cause cause security issues or break programs and also namespaces don't cover everything Linux right so first of all they're not 100 percent right it's nice to have them they help with the security you can break out of them but then there's also things that are not covered in Linux that are not covers

fundamentally by a namespace so just because you know you're like yeah I've got these six namespaces and I'm separating these things but what about all this other stuff that fundamentally you're not separating right that's why again we go back to the m's because you don't have that issue with VM you're not sharing kernels you don't need to worry about separating spaces right or that kind of isolation and then also there's a problem with imperfect design and coding right so I mean who knows how some of these libraries or utilities or even the kernel all these things that are shared in perfect design and coding right increases the attack surface namespaces leaks are present in many

containers and many containers have had little security analysis right again technology somewhat new it's been I mean docker came out of 2013 PCF stuff has been around for at least ten years I mean I think 2005 not mistaken so it's new ish but it hasn't been around in terms of really commercially out there I think you know those of you who put your hands up if I asked how many of them how many of you have used it for more than two years can I see how many of you have used it for more than two years one right so most of you who put your hands up I've only gotten into containers recently including myself right I mean I've only

been using them since 2015 in that sense not now not counting true this containers because true ders it's sort of the the grand great-great-grandfather of containers but it doesn't provide what we would call containers I mean but yeah if you count that of use containers for a long time but not counting true I think most of us have it um and also Dockers image verification is not a hundred percent I said they do image verification but sometimes it's not done properly you can go read articles about it so it's kind of they kind of make you feel all good about it but then you read up means like actually we are when they're downloading images the verification sometimes it's

it's not done as it should be so again shared kernel and evolve it'll be the internal can impact all the containers and operating system kernels provide much more functionality than hypervisors right therefore there's it's a larger attack surface you know hypervisors provide a very specific set of functionality so the tax stuff is smaller kernel is supposed to be a general-purpose that means for the operating system there's a lot of different stuff that it does therefore it's got a greater attack surface and kernels will always have vulnerabilities and containers and containers directly expose it to programs right and containers were not really designed to contain your security issues as I used VLANs as an example or even back in the

day I might be dating myself but in hubs about to switch tabs right you know we're like oh hey we have network separation but that wasn't really Network separation it was a byproduct that you couldn't see the other packets because now I'm gonna switch top two people are talking to each other but as we all know you could do our poisoning or a number of other things to really break out of it right so so again this is we're in the same sort of situation at same point in the journey with containers a couple other things I want to mention are covert channels and defects and harbor protection mechanisms these are relevant not just for

containers but also for virtualization in general but you know there have been a number of cases where people have been taught to containers that are not supposed to be able to talk to each other have been able to talk to each other through the use of corporate channels right so that provides the capability to transfer information between processes that are not allowed to communicate through policy or you've not linked them together so it can be done through storage channels or timing channels so you can communicate by modifying a storage location and then the other the other containers watching that storage location and then you know you they can pick up what is the the message can be

transmitted I mean of course it's it's fairly slow to do that or through timing timing channel so you could perform operations affecting response time observed by the receiver so you performing some operation and the receiver is watching something happening and that response time can be used to calculate the message or decipher the message that you're sending across it's difficulty difficult to completely prevent the issues with covert channels on the same processor because as long as you're on the processors are running on the same processor they could technically observe these types of behaviors right one example could be like how many times the file was opened right and that's that could be a bit from a perspective of trying to figure

it out if this is a normal behavior or not while opening a file is not abnormal behavior but if you were intending it to be as such and observe the pattern of file opening then it can be used to transmit messages across when you're not supposed to be doing that so that that's that's there it's difficult to catch them so techniques for locating potential covert channels or analyzing the resource of a system or doing source code level analysis but again tilde note this looking at the file opening example right to try to analyze well you know okay let's look at all the files that are being opened and see if there's something funny going on well how can

you tell whether these files that are being open are being done so maliciously to transmit some idea across to another container so it's difficult the possibility can be reduced by careful design and analysis right so that's the best we can hope for and then there's the defect at Hardware protection mechanisms so computer hardware is pretty complex and some defects can't have security implications right so there was a row hammer bug that was used to gain kernel privilege right so you can enable changes to values and other programs or kernels so this the real hammer buck for those of you who remember it was a couple years ago it happened by the one process changes the

voltage repeatedly to a row of memory cells and then the electromagnetic field induced was the other one was able to observe the changes and then you know pass the message across right so you can go read more about it but that type of issue with the D Rams and I think Google we're the ones that were able there was a team from Google I can't remember the name of that team project zero I think it was called can't member the name right now but they demonstrated that explode on a variety of systems so so yeah that can't be done let's look at some known vulnerabilities because there have been quite a few no involvement abilities

with these technologies so I looked them up on you know C CDs and there were thirteen vulnerabilities identified today against docker and I don't know if it's visible or not from there but you can see there a number of them so the more recent one published on 2017 no 302 so like pretty much in a couple of what I'm worried about is a today the thirteenth so not that less than two weeks ago it was the race condition and docker engine before 1.12 six which might allow local root users in a container to gain privileges by by using peat race to access file descriptors of a process launched or moved into a container from another namespace so I

mentioned that earlier when we were going through the namespaces I mentioned the peat race vulnerability this is also known as the on entry vulnerability and then there been quite a few others there's often in a range of threats everything from code execution to information disclosures we can hear there's gaining information getting privileges by passing something over flowing code execution denial of service and that gives you the thirteen that they come up with and like as recently as this month so you know don't let someone tell you that these are perfect and there's no problems with them because there are still problems with PCF 17 vulnerabilities identified today most recent ones that I just pulled a

few of that more recent ones for the discussion here potential to subject the universal account I don't think allocation or UA a Roth clients to a denial of service attack there was another one credentials returned from a service brokers from service brokers are logged in the cloud controller system component logs and then are written to disk and often such a log aggregator provide syslog there was another one where it allowed attackers to gain privileged privileges by accessing the UA logs and then subsequently run a specially crafted application that interacts with a configured sam'l provider and then other vulnerabilities as well so you can go to the CDE repositories and have a look but there's ones related to bypass elevation of

privilege etc etc so okay great all these vulnerabilities all these things how do you how do you security use containers how do you contain the potential harm right as the these are supposed to be containers so some of the things you can do here you can group containers on a given VM based on the classic segregation principles right how did you how did you segregate separate your network right back in the day when you were building a network you know you didn't just build one flat network and put all your users in it right you could have done that some companies do that actually lots of companies do that but you're not supposed to well you're

supposed to do in proper network design you're supposed to go hey these five users kind of have the same requirements they're my accounting department and they're dealing with financial data these ten users are my careless salespeople that just click on any button and download anything and then these ten these ten machines are my servers and so based on the what function they were providing and the sensitivity of the data you would take those users and segregate your network right the same classical principles of segregation can still be applied to containers right so what you could assume is you could assume that the container boundary is not secure so you could say okay VM is my separation

boundary given that you know so what do I do to put these things together so you take all your application services micro services whatever they may be and apply the same principle group them okay these six are handling really highly classified data and if they broke its their high impact and these five over here they are not handling sensitive data and the programmers that created them were careless I'm gonna put them together so you that and then you know obviously systems like darker have mechanisms that allow you to you know apply tags when things are being deployed through your dev pipeline and put them in different clusters if you're using docker swarm for example so there

are mechanisms and with with PCF they offer the elastic clusters capability right so that you can create a cluster so you could take six VMs and put them in a cluster and that's called cluster one and you can take another ten VMs and make cluster two so when you're deploying a container you can say hey this container is highly sensitive it's gonna go to cluster one this one is not sensitive it's gonna go to class or two right can automate that through your death pipeline but don't forget he what just the principles you would apply again nothing new you know we've been doing this kind of security separation for a long time creating security zones for

various things you know like network segmentation comes to mind you can apply the same principles here using a risk-based approach I'm considering the impact and likelihood of that particular application and services obviously should be fun as I'm privileged there's nothing new right any of you have been doing little system administration assists admin type work you would know this and you should try to not run things as routes and privilege should be dropped as soon as it's not needed so if you do have something that needs to escalate its and get privileged then you should try to drop that as soon as and that job is done and and you should treat root inside a container as if it

was rude outside the container and you should only run containers from trusted parties right so don't just download something and you don't trust because who knows what its gonna contain again that's nothing new and in this case and you should follow layer defense approach so you should use a farmer or selinux in addition to all the other stuff so that's the mechanism for supporting access control those security policies right so just because you separated the containers you like okay I've created separate security zones and I've separated them that may not be enough right you should still follow the layered defense approach you know configure a primer configure policy set comp all the other stuff right security

is all about doing as many things as you can and all the different layers and hopefully if one fails you've got something else that that will will help prevent the attack from taking place or being realized what else can you do you can standardize and verify hardened host OS you can scan containers for vulnerabilities as you're rolling these out through your dev pipeline you automatically you should scan these and that's one of the nice things about containers than one of the DevOps models that people are going with these days is that you you you deploy these often right so in the old days we had a physical box we installed Linux on it and we saw this heavy you know Apache

and my sequel and all the stuff and it was like heaven forbid we have to go patch this physical box it was a nightmare right but now you're just putting these containers I've read every day every week you're just you know you're deploying a new container you know something dies even on the spot you're your scheduler my redeploy container so that's an opportunity and these things are all virtual so that's an opportunity for you to rescan to patch and to put all these things back in there and which we didn't have that opportunity with with the old physical system model so you can you know you scan for open source modules licensing now we're correct configuration and all

those kinds of things you can configure through your build pipeline and you should measure the containers and sign them right so you should analyze the sandbox containers build a profile you should confirm signatures at boot confirm host OS integrity you should monitor and detect anomalies behavior you know all that standard stuff alert log or try to prevent and you should analyze the users logs to see if you can identify any weaknesses right so if you know if you can adopt patterns if you can share and learning across different instances and and things like that so in conclusion containers are good right we want to use them I'm not here to tell you that hey there's all these security problems to

runaway container there's a lot of uses and we looked at a whole full slide full of why you'd want to use containers and I you know I think a lot of you realize that they help ease software management so lots of value in using them but just got to do that security so work is being done to address the security issues implementation of user namespaces is a big one in the recent year so that wasn't theirs that was that's been a big leap forward for security but this should be used with caution right so even with the namespaces that are implemented you can still break out of them there are processes in the container should not be given privileged access

they're good for deploying apps that are trusted so from the same vendor other mechanisms such as SELinux Tech comp app armor and separate user accounts should be used in conjunction and I think I'll wrap it up with that and all open up open up the floor for Q&A there's anyone that's got any questions hopefully I'll know the answers that I don't know all the answers but please shoot there's a question back there I have not used it actually no no wood

docker unfortunately I won't be able to give you a good answer on that big docker offers a bunch of good tools on their website listed I'm primarily using PCF and we're not using any of those in our environment so lookup Dockers website they've got a list of tools that are used for that purpose but for cloud foundry I actually should probably look it up because I need to actually start rolling that into my own pipeline but I don't know the answer to that yes just starting actually yeah if you want to we're just like jumping in AWS and we're kind of trying to merge what we've learned in PCF and bring it to AWS and

see if there's any value but can you elaborate on that because I know you're an Amazon expert so okay what advantages does it provide okay would lambda yeah

so if you're trying to do like a micro service that's probably the way to go because you don't have to set up all the heavy like all the other stuff that's involved inside of nos and putting this micro service in it you can just put it in lambda and they little is that how it works yeah yeah yeah we're just we're just exploring that I think we're going to be going in that direction

so what do you mean like in terms of I just want to make sure that I understand the question so in terms of what you're deploying into your container or in terms of how its configured cuz I mean there's there's quite a few different places where you could do that right I mean you can enforce the policies with a farmer you can do the set comp policies depending on what you're trying to target right like a visit is it an application access control which you can accomplish with with app armor is it access to resources is it pipeline yeah yeah like to check to see if you've got certain things in place or not like we

do that sort of thing like we're using Jenkins right in our environment so through Jenkins we enforce we've got we basically configure a baseline like what good looks like and then it's like if you're deploying it then it should have this as this configured it should be like this is this this so when you run it through that pipeline it looks for that and if it's falls outside of that little flag it uh says this doesn't fit so not accepted right is that kind of like what your profile the application of what it can do or cannot do yeah yeah what the changes that have been propagated so if there's like a new thing you got to add to your why that we

do that all of our through like we do that through our configuration like through our build pipeline we manage all that configuration stuff through as I mentioned through Jenkins but that's what we yeah we have a configuration repository we configure what should be a man if there's a change in policy or like like whitelist in your example we would just put that in the configuration repository and then that enforces it on the pipeline and it deploys it as such that's how answerable you could do that as well yeah yeah yeah yeah good any other questions yes sir

we're not so I'm not using docker I'm using piece we're a cloud foundry shop where I move but it's the same principle so I mean we have you know standard config we baseline a bunch of these configurations right so when we're rolling these things out like we just check and see does it fit that baseline is that it doesn't fall outside that baseline like what the developers built right so we say these are the things you should have when you're deploying this container should have ABCD these policies should be enforced SATCOM should be like this etc etc and if it compares that in the developers doing something else I cited that it'll fail so it won't roll out that container so

you could build that into that's that's part of like depending on what Bill pipeline you're using you could do those customizations within your bill pipeline yeah that side yeah and even with AWS like right now we're going in the AWS journey but we're trying to do the same thing because it's all about automation right cuz we've you know we've come up with a you know we had a workshop two days we sat around we figured out these are all the things that we should have in terms of security controls right for AWS but we don't want to manually go and enforce those I mean it doesn't make sense so we build an automation through scripts so that someone's deploying it

it'll look for all those things to make sure that they're in place you know whether it's a whether it's a particular version of TLS or whether it's some something that maybe you should have like mutual authentication between your whatever the configuration may be we usually enforce that through our our bill pipeline yeah yeah cool any other question okay well if there are no other questions I thank you very much for coming thank you for coming to besides hope you've had a great first day there's gonna be more excellent talks tomorrow and and yeah enjoy thank you [Applause]

you