2018 BSides Toronto: Fernando Montenegro

why do I feel like I'm in a monster truck rally all right if you could take your seats our next speaker is about to get going here the next talk is called evolution of container security what's next and if you could please put your hands together for Fernando Montenegro

thank you very much everybody it's a true honor to be here and especially thank you to everyone who's new coming to besides this is a wonderful event I hope you have a great time here and I hope to see you speaking next year here I had a few objectives for this talk I want to talk a little bit about where container security is nowadays I want to present it in the in a broader context I want to touch on a few key market trends and I want to discuss what's next just as a point of introduction I've been a member of the security community in Toronto for about 18 years I'm now an industry analyst at the 45 foreign

research I've been in sales engineering roles before and and whatnot and I have I have a small disclaimer first these are what I'm presenting are my opinions as an analyst as I cover this space which basically means I reserve the right to be woefully wrong about what I'm talking about I will mention tons of vendor names that's not an endorsement of those vendors and to the best that I can do I try to make lists as comprehensive as possible but in in most cases especially container security I end up missing quite a few so the list is more representative and not comprehensive so I mentioned I'm an industry analyst and I wanted to talk about the

methodology that I use to build this presentation for 5-1 research has a long history of the research firm there are two major sources for this one is we run what's called the voice of the enterprise survey which is a long-term relatively I mean serious study on survey data it's not your typical five bucks for for an answer kind of thing we do have strong methodologies about it and we're and on a yearly basis different quarters the different quarters there are different set of questions the other thing I've used for my for this research is just my time as an analyst talking to customers who have inquiries with us looking up in the pen doing independent research on

vendors and and so on so what did that research show why me why am I here talking about the containers because of this so this is the data from this is a 20-18 Survey data it just came out and if you ask when we asked people who are what's the the features they're using the most containers came out as one of the highest usage features in in public cloud deployments right this is this is particularly interesting because containers are something that to some extent they've been used in different ways by different people we also ask them what is it that they are planning to do in terms of features as they roll up a thing prove they are their presence

in public clouds and as you can see their containers takes a very strong second spot right after machine learning right and to some extent the machine learning component is is something that a lot of the larger providers are offering as a service right there is specifically offering the service as opposed to having customers spin up their own tensor flow or Cara's pipelines so you can you can say that containers are the highest the most interesting footprint that people are going to be deploying in public clouds now why is that and the best that we can come up with is based on this question like how do as people approach cloud native applications as you are building

as your company is building an application in the cloud what it is that you're looking to do and the top answer is you want to design it to run effectively on any cloud environment and it's this portability around containers that makes it so attractive for this use case it's the notion that and we can debate this I'd love to have the conversation around whether we're doing how we spin up workloads in multiple clouds but the fact of the matter is containers are an incredibly portable format for doing things now one thing I don't want us to ever forget if that containers came about for a different reason containers came about to resolve this problem right this is why fund I'm an

infrastructure guy I love networking infrastructure but containers came to mind because we wanted to hey it doesn't run and it doesn't run on prod but it runs fine in my environment so it is fundamental it so a lot the container image for it containers the definition are it's an image format where you can Patras where you can pack dependencies and it's a runtime that's going to execute that so we've kind of gone away from we've kind of gone away from this theoretically to something like this right a rainbow of containers containers are going to solve everything in in in technology and it's not quite like that there's a lot there's a lot to containers that that we don't that we

still have to work through I came across this cartoon a while back that that I think is something it's something I think about all the time right the reality for most organizations if that migrating a legacy application to a cloud environment is much more like this then hey I'm going to just spin something up and off we go right and what this means is that discussions around container security are much more interesting in terms of understanding where the container security technology fits within a much broader lifecycle of things so what I wanted to present was how I frame my research in terms in terms of container security in terms of a container lifecycle before we get to

that though just a quick recap container technology has been discussed significantly over passed a little while but I wanted to do a refresh and mention that in fundamentally what we're talking about with containers is there is an image format where people packaged their dependencies those dependencies then that image format then gets put into an environment where there is some sort of of runtime that unpacks that environment and execute that code right and there's different there's a difference between the Linux environments and Windows environments in what the subsystems are in Linux with all or many of us have talked about things like namespaces and NC groups and capabilities and so on which is basically the way that Linux

will and I have more of an in Unix Linux background so that's where I'll focus on but it's fundamentally how the runtime is going to be able to segregate the information between the processes that are running on one container versus the processes that are running on another right and Microsoft has announced has had container support in Windows since 2016 I believe Windows 2019 is just about to be released and that has built-in container support as well I did not include 20-19 slides here so understanding when we talk about container security a lot of times we spend with a lot of the material I see we spend talking about okay so how do C groups work how does how do we do how

this selinux work app RM or in those things but there's a much bigger story around containers so at the risk of simplifying it a little bit too much I wanted to talk about what do I see as a pipeline for a container image right you start with an I'm gonna call it an artifact right I mean something that lets say you download an image from somewhere let's say it's docker hub let's say it's another sort of repository that you had access to or you create one but let's say to download theoretically what should happen in most places is you download that into your CI CD pipeline so your building code you're building that you're bringing the

dependencies into your pipeline that pipeline is building the new version of your application of your container then that gets pushed to a container registry where it's going to be instantiated later on and of course I'm glossing over a ton of material in terms of dev tests and proud I'm there's a lot of things going on here unfortunately sometimes this happens as well people just download images off the internet and just put them on their registries and they should really not do that once they have that that container registry then when you're going to execute that you're going to have a runtime environment so that's basically where your your your docker run time is running your other

runtimes we'll talk about that later that are going to and that is going to may be made up of the environment where that container is running so how that role that the host is processing containers as well as the workload that is running itself so when I am thinking about what is a specific container doing and then that's where you pull the image from and that's your your container runtime so I call this the build phase the ship phase and in the run phase now one of the things we should keep in mind and I've adhered to the next presenter you cannot are realistically most large or most production container deployments have an Orchestrator layer on top and

that I will put in the data typically put in the ship phase as well so I want to talk a little bit about build time considerations for containers I'll just skim through this a little but during the build time what you should be thinking about as security professionals is fundamentally application security how all about the code that is actually going into that container you should be thinking about how you scan the images as you are building the image when that is when you're creating that that image for them for the that you're going to push out later on how are you doing vulnerability management on those components you don't want to import an old library that doesn't make that

that's got ton of vulnerabilities by the same token you don't want to import libraries that may break your policies maybe you have a license restriction you cannot run a GPL code for example right so you have to have a layer of checking for that there and the issue I like to say is realistically issues should be treated as defects on the pipeline so that your development team can just process them going along after that you sign your image one tip I like to comment on if the this notion that ideally it's at the build time for your container that you're going to reduce the size of that image and one of the ways to do that is something called

multistage builds I have a copy of a file here basically you start with it with an initial image from the first image builds the minimum libraries and then you build your application and then you copy that into a very small container some of the examples I don't have them here but they talk about how you can reduce while an image might be 300 megabytes in one traditional way you can bring it down to five or ten megabytes sometimes that is the build time considerations when you're talking about ship time considerations there are two things that people should be considering one of them is on the image registries themselves and that the topic here is again you should do for their

ability management on images that are on your registries because you may build something today you may push it to your registry today and you may not use it for two weeks and lo and behold it's somewhere in those two weeks a new vulnerability pops up on that image right auerbach stands for role based access control i'm using that as a catch-all phrase to refer to what permissions you are giving people in the organization to ship images back and forth and the other ship time config consideration is how you secure your Orchestrator there's a wonderful session coming up in a little bit about kubernetes security so I'll leave it I'll leave that to Hayden but you are

there are different considerations and we will all share the slides later run time as I mentioned there are two elements one of them is how you do host protection how you secure the workload how do you secure aware that your container workload is going to run and then there is the container runtime itself I'm the host protection is your traditional how do I secure server you use CIS benchmarks to harden it you use firewalls in in the access network access control to block the hosts for the container runtime you're a major consideration then becomes what exact actions is that container taking right so you can analyze system calls you can segment the network with with net

network level segmentation as well I mentioned here briefly alternate runtimes one of the developments that have happened over the past few months is that couple years is that there are a number of different runtimes for a docker for containers that each have different pros and cons in terms of security features right some of them segregate based on VMs some a great segregate just based on on system capabilities taking a quick look at some of the trends that we're seeing in industry right I think that one important trend as you look into container security is that there has been significant governance efforts around container security right this released a complication container security guideline in September of last

year that that's that's quite informative the Center for Internet Security has as it says here docker and kubernetes benchmarks and I put the asterisk here because I think there's a CIF benchmark for pretty much anything include and including all the the large cloud provider environments where you're going to run things the open container initiative is the effort that defined the image specification and the runtime specification for containers so as organizations create new container runtimes they are usually building them to OCI specs and the cloud natives computing from the is the one that mostly responsible for the it's a very large effort responsible for kubernetes and specifically contain their networking the container networking definitions it's coming out

of the CN CF and both o CI and CN CF are under the auspices of the Linux Foundation a couple of points on NIST the the NIST guidance it's it's a very easy read but the key points are and I find it interesting the one of their key points is fix the organizational aspects so here we are on a container security technical document and they are telling us to pay attention to the organizational aspects of how these things get built right there's a there's a little bit more guidance around containers specific try to use a container specific host versus a generic host because the footprint is going to be smaller the the attack surface is

going to be smaller and a couple of other things but also where possible think about updating whatever vulnerability scanning or run time scanning tooling you have for containers versus using a standard default I mentioned the CNC F I'm not expecting you to be able to read this these are all the projects that the CNC F is responsible for this again you'll get the slides this is a great overview of pretty much anything to do with cloud native technology another trend which I'm seeing significantly is what I call the rise of the managed runtime so this one stands for as your container instances this is AWS Fargate this is google kubernetes engine and they just pick the three top

ones that the other cloud providers have them as well but this is the notion of you as a developer you don't need to worry about the hosts runtime anymore really what you should do is you give them your container definitions right and they will orchestrate it for you you don't have to worry about if there's a vulnerability on the backend or not on the on the host environment and this is a significant thing from a security perspective because now you no longer have a host to work from right so what ends up happening is that you have to start instantiating security within the container itself versus having to rely on a system based agent another

trend I like to talk about is that there are a different commercial approaches to container security right there's two slides the first one I compare these are all the the cloud native containers specific vendors twistlock and aqua I tore the most usually recognizable ones and they come with the benefit of having very specific very rich container security features with the downside of hey it's another vendor that you have to manage right on the flipside the names we have here a black duck - ops if trend the cloud passage koalas rapid7 these are some more established vendors in most enterprise environments and those vendors have rapidly adapted they are tooling to support containers as well when I say adapted I mean that they've

they've added container functionality tenable for example for bought the company floor check a little while back but what they are doing here is that they are doing this by adding the functionality not to a level where the these more specific vendors are so depending on the tooling that the depending on your on your needs you may do just as well with your container security with your existing enterprise security vendor for some of the container security features right the other thing I wanted to mention is that especially in enterprise environments it is unlikely that you are going to be deploying containers in production by pulling images and instantiating things yourself it what's more likely to happen

is that you're going to leverage a container platform Red Hat has openshift docker has an Enterprise Edition version pivotal has the difficult foundry and and I mean mesosphere has DCOs Rancher has venture labs and what these tools have is that they they have a broad set of services of security functionality again at the downside of potentially having limited support for new features and what they all do here is that they will then typically these companies will partner with the with the newer ones for some of that more advanced functionality I also have here the cloud providers themselves because they are adding containers security functionality themselves right and one of the things for example that was just

announced at Google next Google now have support for what they call binary authorization which is they themselves are going to lock you from running unauthorized images on their kubernetes environment so now you can start leveraging what the provider themselves is doing the downside of course is limited portability so we talked a little bit about trends where where do I see container security moving forward right one of the more interesting developments that I think are coming is the notion of a service mesh right and the service mesh is a higher-level construct on top of a an existing environment that is responsible for things such as how do you route traffic between multiple containers that are running how do you do that and that's a

tough problem on on kubernetes in containers which is how do you do how do you observe traffic like how do you do debugging and so on and service measures have a security element as well they will bless you they will have a they have traffic encryption usually built in for a mutual authentication via TLS and they will have some level of some level of auditing and fine-grained policy this container or this pod can talk to this pod elsewhere and this here is just an example of the SQL framework it we can go into more details later if if need be where does this fit into the broader theme of of that vision that of this framework that I use I see service

mesh fitting here it's a way to control that runtime or you're going to run so you're going to be able to it doesn't matter where you're running you can have a service mesh that's going to distribute it across right the other thing that I think is important from a container security perspective to be aware of is not it is not a container itself it's the notion of event-driven functions a service which people sometimes shortened to there's a discussion of whether that is service or not all right I'll consider it I like to be a little more precise and with if you're not familiar with this concept already it's the notion that you're going to be

not writing anything else other than the business logic for your application but you are going to take the you are going to take the services you're going to tie together services from the environment or your running to do whatever it is you want to do it's a fantastic example lambda is is one of the more popular ones but sure and Google have them as well and there's all there's other efforts open risk I believe with IBM's and there's other efforts for doing service but fundamentally they all revolve around the same thing you're writing code you're no longer managing infrastructure right there's again there is no host to monetary from and security then becomes an application security and

there I say an anti-fraud conversation more than an infrastructure one so I just want to recap we were going to talk about containers in context we did I talked a little bit about our adoption I talked a little bit about this notion of build ship and run we were going to touch on some market trends I did add something around the governance competition and new services that are coming along and we were going to talk about what's next right so I think service matches very interesting as I talked a little bit about recommendations now so what do I want you to leave this presentation with number one is that don't talk about container security in the context of the

runtime alone don't talk about container security in the context of container escaped from from a host environment there is a much broader context on which container security falls right one suggestion is that as we as people move to rent times that are more that are more independent like those manage rent times don't pay so much attention to that host environment pay more attention upfront I think that what you're going to find out as you look into container security is that a lot of the benefit you're going to get will be from working with your developers to build their to build that security into their pipeline and if they are not using CI CD if they are just

downloading images of the Internet at least doing some level of sanitization on the images that they are pulling from if I had to suggest how you would split your time I would suggest something along these lines fifty to sixty percent of your time should be on the build phase twenty to thirty on ship and ten to twenty on Monday but again reserving the right to be wrong I'll I'll leave you with some very interesting projects they're usually open source around these areas from the ability of scanning hardening monitoring and a couple of others and these are the the sources of information that I go through on a regular basis for anything to do with container security right there's more of

course but this is a this is a really good start I'll leave you with with with one idea that's that that's not mine Kelly Shortridge she's a she's a security professional I have a great grasp and on DevOps and security and she argues this was just recently DevOps is actually in fact best friend right and she's right right I think that we're trying to do a lot of the same things we're trying to get code to run the way we want it to run and no more than that we're trying to get the we're trying to minimize the impact on the on the environment and this is a great way of doing that I will

leave there's my twitter handles people have want to discuss publicly I will be around later for questions as well but for now thank you very very very much for your time and thank you [Applause]

aidan to the stage Hayden please come to the stage