You Might Still Need Patches for Denim

welcome everyone right now we have in our next speaker is up and so this is second of three container talks so you might still need a path your gene for your genes but you no longer need a patch so take it away thanks everybody for coming my name is Dan I'm an engineer on Google Cloud I work on a whole bunch of open source tools related to container security like I said almost all this is open source you can find me on github and kubernetes the mini queue project and other things like Kanako's scaffold I'm Ann Curtis and I'm Mike atrocity I'm a product manager in security at Google and I work on container security yeah so

before we get started I asked the audience a few questions to gauge familiarity could everybody here put their hands up if they are using containers in production Wow all right has anybody here heard over tried out kubernetes and most of the audience cool yeah awesome I'll keep your hand up if you're using it in production half of that point if you went down here cool all right so thanks everybody for coming in this talk we're gonna cover what containers and kubernetes are because you've heard of them but about half of you have actually used them in production but this talk is not about how to secure kubernetes they're a bunch of talk specific to that at other

conferences instead we're going to take a step back from kubernetes itself and talk about some of the implementation details like containers and how they affect security more generally we're going to talk about software supply chains and how containers affect that I will describe what they are and then what changes you have to make to them when you move to containers we also have a pretty exciting a demo I'm halfway through to show how all of this works both with software and with these jean jackets and then maybe by the end you'll believe us when we say that we can come micro services and containers can actually help improve your software supply chain strategy so first of all

containers kubernetes and how that changes the model for security containers are all the rage these days but what is a container the common definition is that it's a packaging tool in a Linux kernel comprised of two primitives called namespaces and seekers but unless you're a kernel expert that probably doesn't help very much in this case the physical metaphor or container really does make a difference shipping containers and the whole shipping container supply chain revolution is build up on the idea of standardizing the format which we ship physical goods putting all these into containers that are the exact same size and shape a lot of supply chains to move goods around the world a lot more

efficiently and that's what containers are for software as well containers standardized format for packaging your code as well as all the system level dependencies into a single unit that can be shipped and deployed and run anywhere that implements that container contract there are a lot of parallels between containers and virtual machines but the main difference is around resource isolation virtual machines provide isolation at the hardware level whereas containers require you to run everything all on the same host in the same kernel container just have a lot less isolation in the sense but they provide for a lot better resource efficiency the support for containers was merged into the Linux kernel sometime around 2004 but they

didn't really take off until recently when docker exploded in popularity dr provided a single tool chain for building testing and running them simply there was batteries included and that was installed on a wide range of Linux distributions many people here have already tried out kubernetes and are using it production so we won't get too far into that but kubernetes provides a set of primitives for running containers across a cluster of machines those could be vm's or physical machines it defines a set of primitives running scaling monitoring logging and doing all the things you need to do for production with containers containers provided the magic of encapsulating your application all of its dependencies into a single

unit and then kubernetes lets you run that across a large cluster from a security point of view looking back at the year and threats we haven't actually seen that many active attacks on containers in the wild it seems that the industry is still just kind of figuring out what these things are and likewise attackers also figure out these things are and how to best exploit containers and kubernetes so in February of last year twenty to twenty eighteen Tesla me the news for attackers finding and accessing their communities dashboard and using cloud credentials that they gained there to mine hooked our currency it was really similar to an attack that happened on Aviva in late 2017 in May Shopify had a

researcher report of ulnar ability in their cluster configuration this was thankfully not exploited in June Weight Watchers had the exact same issue that Tesla had with an unsecured dash board it was also discovered by a researcher and not avoided and the same month several container images were found on docker hub that had embedded malware for mining cryptocurrency what's happening more recently well we don't even really know about that yet maybe the NSA does what's really interesting about all these attacks that you'll see is that none of them explicitly attack containers or container eyes were closed application dashboards being left expose has been a problem since at least the eighties some of these do attack misconfigurations

though in kubernetes but the end goal isn't even the application that you're running in kubernetes it's just a compute right that's not how attackers make money they want cryptocurrency mining they want botnets they don't care that but the fact that you're running communities and so I hear security people complain about containers and communities all day long in fact that's kind of my day my day job some of the typical issues I hear are first a lack of understanding they don't know what containers and communities are and are afraid to learn now you know a little bit so maybe this is less of a concern after all it's just more for you to secure right frustration with existing

tooling of mechs since we're already dealing with an industry since we're dealing with industry that's still relatively new and where adoption is still rising and there aren't necessarily the same security tools that you're used to so the idea of an IDs or an application layer firewalls or kubernetes are not really real yet so if you're walking around the RSA expo floor and someone's trying to sell you that you're really falling through a snake-oil right now next up a common concern in containers container escapes a lot of security professional professionals are worried about using containers for this particular reason it's if we were running a multi-tenant environment and one workload can escape its container and affect another

workload in the same environment however the the impact of and the effect of a container state can be quite high however we haven't seen many vulns like this in the wild and it's not really a surprise that containers aren't really good at containing from a security point if you're right that wasn't the original intent so you do see projects now coming out like jivas or contact containers and firecracker that are supposed to try to address this explicitly lastly and you have come to realize with these issues it just doesn't matter in some sense right a lot of people I talk to it's their boss's boss wants to use containers the dev team is gonna start

doing kubernetes and now it's your job as security people to secure it you don't have a choice so you've gone through the five stages of grief and that's where most of the conversations I have are which is I have to use this thing help me secure it please help me and help please help make it better the situation so there's ongoing work in all of these areas better education awareness tooling isolation etc but that's not so much that last piece so today we're going to try to convince you that when you're dev team says hey we've been playing with kubernetes you should be actually being excited about that because I means you could have a lot

better security in a lot of different ways in a way that was just never possible with your previous your previous infrastructure a lot of what's needed protect containers are constructs that were already familiar with right us--or gonna want things that look like firewalls access management audit logging ids/ips and more here are a couple of areas that are actually slightly different for containers in terms of security first as a surface of attack container bundles binaries in libraries as part of the container image so you don't need as much in the host image if you're using containers as intended this host OS should be really small really minimal it doesn't have as much blow which then reduces the surface

of attack next resource isolation this is built into containers allowing you to easily allocate resources like storage volumes to certain processes using namespaces in terms of permissioning containers have normal access controls as you expect for controlling access to shared resources containers also encourage the use of micro services which enables fire in a grained access control to read to those resources on the flip side containers are by construction much more powerful and have a wider range of syscalls that they can perform which opens up a bigger attack service to the hosts lastly lifecycle containers enable fast development but I also and you know are frequently redeployed and are fairly short-lived but that often that also offers the opportunity to have a more

structured automated software supply chain with compliance built in and we'll talk about that in a second at the same time if an attacker ends up in your infrastructure that container might be gone before the time before have a chance to actually investigate into forensics on what's going on so it's kind of scary right now we'll talk about a traditional software supply chain before we talk about how containers change that so if you're working in IT and a large traditional Enterprise today might not have even heard of a software supply chain a software supply chain is not how you get your finance team to fulfill the invoice and do procurement for the software unit and it's not how

you prevent on the one hand hardware from being your data center how a Bloomberg that's a hardware supply chain a software supply chain is anything you depend on for your software it's all of your binaries all the source code used to build though of the open source packages your dependency manager pulls in and that code your vendors vendor wrote it's even that kubernetes dashboard you forgot was running in left exposed that's all of your software and how it got there I'm so traditional patch management and a large enterprise today looks like this we've got a couple steps the first step is to get the patch this is gonna come from a variety of places it might come

from a stock or a distributor you peg it might come from a mailing list some open source project or it might come from a vendor patch feed that you paid for to keep it exciting sometimes this patch hasn't even sent to the dev team sometimes it's sent to whoever purchased the software it's you have to chase this down across your organization next you have to apply that patch and figure out how to first you probably want to do this on a test server since you don't have a test environment so you pick a workload that isn't too critical and apply the patch there and see if it still looks like it's working if it does

work then you apply the patch everywhere else at least everywhere that you can think of you're gonna start with that spreadsheet that somebody filled out a couple weeks ago and probably hasn't updated since then of all the services running in production after that works you pick a couple more servers at random and just keep applying the patch until you think you found everything but reality is that in a couple years from now you're probably still gonna find things that are patched that the cryptocurrency miners have been running out so what's wrong with this setup well first of all it's a pain to maintain you might have an asset inventory which lists all of the hardware you have and

the version of software that you think it's running and even if you're using a tool for this it's basically a spreadsheet right there's all there's a lot of manual checking I need to me patch you have to go manually update that tool you can't dynamically create this easily which is annoying when you do patch you take your servers down this is some poor soul up at 2:00 a.m. on Sunday who has to manually go look to go hug a cable into every single servant up line update a friend of mine was telling me the story and you know being a Bay Area tech company less than five years ago of realizing at 10:00 p.m. that his

update was failing and both you know not having enough caffeine and not having a nap deodorant to quite make it through to the morning so to actually get that out there that's a pretty terrible situation and I'm sure it's very relatable when there's a new ode a vulnerability you have to quickly apply a patch and it's also kind of terrifying even something like Spectre a meltdown most large corporations were unable to patch before their next regular patch cycle if they were even lucky to get into that pipe pipe cycle leaving them exposed to potential attacks but lastly the number one question that always ends up getting asked is am i affected do I have something

somewhere in my infra running with that unpatched fall after all that's the first question that your seaso is going to end up asking you and so what does an ideal software supply chain look like well running containerized micro-services leads to a different security model programs let's the air command that we're trying to make you have to adopt a fundamentally different mindset which we'd argue actually leads to better security long term containers are short-lived immutable and frequently redeployed which leads to a few nice properties first containers give you a software supply chain because containers are frequently redeployed you can easily see what ends up in your infrastructure and you have an enforcement point to control which should actually be there

continues let you patch continuously and automatically rather than pushing a single line of code change you rebuild and redeploy the whole image if you have other processes that can take place passively like patching your OS you can be constantly doing these and rolling them out as part of your normal rollout process last night containers mean you can actually tell if you're affected by a new vulnerability since containers are immutable you're not changing single lines of code you're changing whole containers you can look at what you deployed and see if the vuln is there it also means that if something does change in production when you didn't expect it to that's a built-in alarm that something went wrong

so let's take a bit more into each of these one by one all right so now we'll talk about the supply chain that containers give you specifically what's different about them compared to before you use containers well historically with the monolithic application running on a virtual or physical machine when you want to make it change your developers would SSH into that machine or use some tool that basically does the same thing as SSH again they're gonna make those kit code changes or infrastructure changes without really knowing what's going to happen or without keeping track of what changes have been made by other people there are manual adjustments other developers made other changes including things like vm

is restarting and patch to be applied automatically you might not even know exactly what's running on your VM at this point as everyone's just changing things on the fly it's not only hard to debug but there's very little formal process around it next time you have to do another one of these you'll just association to the VM again and hope the same patch gets applied this isn't a great security story and it's really difficult on your ops team they don't really know what's running anymore your infrastructure is coupled to the code running on it and all of the changes that were made to lead it to that point and when you have systems or people touching this all day long

you know what's known as immutable infrastructure which is pretty brittle it is technically possible to achieve the opposite of this immutable infrastructure without moving to containers but it's difficult containers are basically designed from the ground up for with immutable infrastructure in mind containers define a standard pipeline from code to node based on the ideas of immutable infrastructure you can write your code and ensure it meets your desired standards for build testing signing and deployments and then deployed as an entire unit inside of that one container micro services get redeployed as individual containers or sets of individual containers rather than changing specific lines of code and rebuilding binaries on a host directly this is easier to debug easier to

maintain and easier to change it's easier to change the net bag pom from the eighties so with this defined pipeline there are a few different pieces to consider when you start building a container you start at the top level usually the from line and a docker file sometimes referred to as your base image this is usually some kind of Linux OS distribution like Fedora or Debian or Ubuntu plus some company specifically chief applied on top of that next comes your code usually this is the code used to build your binaries and all the dependencies that your code needs both language level dependencies and then system level dependencies after all of that gets built you have your application image

this is the final deliverable that you're going to be deploying to prod once that's ready to deploy you can run all of your tests on it make sure that it is correct and then you can do the actual deployment of that container image always one piece you can also enforce policies at this deployment time they kind of vary between your different environments you might be fine running something that hasn't been tested in QA because that's where you're going to be doing the testing but you want to block that from running in production altogether all of these components the nouns and the verbs put together make yourself for supply chain next my containers let you patch automatically

and continuously easily so I'm really really trying to make patching as I say again and that's because containers are sexy now so obviously patching is gonna be sexy again right even today many security attacks that occur in the wild containers are drive-by attacks attackers looking for deployments with known bones that they can exploit and those vulnerabilities are rarely zero days right there things have been around for years left unpatched like wearing sunscreen scanning for vulnerabilities and patching them is like one of those boring practices you should really be doing the patching containers is different than patching vm's containers are meant to be immutable remember so you as Dan just said so you rebuild and redeploy the whole image when you need

to change something this happens very often as containers are short-lived Cystic estimates that 95% of containers would live for less than a week but wait that's really often if you look at traditional patch management patch man patch tuesday comes once a month you might have some additional patches that you have to apply weekly and you might even have a you know regular maintenance window to actually apply those patches but there's not enough time of the day or Club monta in the world to deal with deployment that only live for a week it's just not worth your time the club bought they feel about their custom so going back to our patch management process what's different with containers

first with containers you don't patch live containers you patch the images that are in your container registry you can easily see everything that has the old unpatched software and apply the patch everywhere at once and also remove stuff that you don't want anymore from that from that registry not allowed into your environment then you can roll out the new images the fully patched container can be rolled out or rolled back as one unit so the patch roller process becomes the same as your hopefully very frequent code roll out process complete with monitoring canary and testing so I guess it's maybe okay if you're still running out as 1% tests to prod as your as your test your patch

that rolls out using the same in the normal process in a predictable way once you've rolled that a newly patched container and tested that everything works as expected then you can move over actual workload traffic this means you can patch your application without having any downtime and right one of the points I was just making that the patch roller process becomes the same as your code roller process this is really critical it means you can deploy things gradually with your entire testing suite and rollback as needed patching is no longer this ad hoc thing you have to manage this will certainly make your essays happy this is in line with one of the key SRA principles that you have

progressive rollouts nonemergency rollouts must proceed in phases and it's specific to your workload an environment that determine what percent you you want an able at each stage how long you spend in each stage etc before you're comfortable with moving to the next stage your team can supervise these rollouts to ensure that nothing unexpected happens and then adjust accordingly an alternative although it's less preferable because it happens on that ad hoc basis is to let the Rope the rollout happen unpredictably ad hoc when containers die so if you have a patched image in your registry next time the container dies Koreans will spin up another container to compensate and any patches that you apply to that will

naturally roll out your infrastructure depending on the life span of your containers that could actually be patched in a matter of days even though it's tempting to roll it roll out a patch for an O'Day as quickly as possible that's not always the best decision especially if you know that it's not yet being exploited in your infrastructure nobody wants to have to make the choice between downtime and being vulnerable to something but trust me it's way better than not having the chance to make that choice agreed so in some sense taking all this together it means that patching is dead very patching doesn't matter as long as you're killing your containers on a regular basis your software will always

be up to date that's a huge security improvement and that's what I mean when I say that containers can actually be better for security and how you manage patching in your environment this is the takeaway so you're skinnier images and finding a ton of vulnerabilities now what practically how do you address all of these there's three main things that you can do to actually deal with that the first category is patching we've been talking about this isn't in you to be doing on a regular basis through your docker build files it's what we've been talking about add app get up to update and apt-get upgrade to apply patches to your images the second category is

removing packages that you don't need a stock DB rebooted system comes with many packages in many cases you won't actually need all of them turn in application having those additional packages on your system creates a larger attack surface unnecessarily again it's not that minimal thing that we were talking about earlier so you should analyze what you really need and if there are the packages that you can be removed then remove them you don't want to collect operating system packages like you did with box the third category is moving to a smaller distribution like Alpine or DD and slim you can run your application using one of those smaller distributions to have a smaller surface surface of attack again another option

here are also a distro list images alright one more quick poll how many people here are responsible for some form of production infrastructure today quite a few people alright great so this is a safe space I'm not gonna tell your boss but put your hand down someone give you one day and you have to give me a list of every service running in production and all of its dependencies keep your hand up if you can do that one person two three people okay this is going to get a good harder what if you needed every service and the exact provision of source code used to build each of those dependencies okay I call you liars it's pretty honest audio

sweet all right so what if you only have one hour to get that information nice awesome you must be using kubernetes alright so this is the real power of containers for security I'm containers are immutable and content addressable so they're stored in such a way that they're addressed by the contents inside of it which can't change this means you actually know what's running in your environment for example which images are deployed knowing what you have deployed in your environment is one of the biggest pain points of traditional patching you couldn't even tell if you were affected by the patch here everything lives in a single registry so it's just a quick check to see what's running and what's inside of

each image inside of that registry so what does it mean from a security point of view it means you have a full manifest of everything running on every production system you can check this is called your container registry things every image you ever built putting everything inside of it just waiting for you to inspect so to figure out what's in prod got a couple steps here here's how its in practice first your container environment usually kubernetes or something else contribute you provide a list of all the containers running this is just a few lines of bash on coop control from there you can inspect the contents of every one of those containers from looking at them inside

of your registry once you know what's in those packages you can cross-reference those with one of the many excellent cv databases out there there are a few gotchas in this process first of all you want to make sure you're working with the digest form of container addresses this is the immutable version we talked about before you also need to make sure you're leaving yourself enough breadcrumbs to work your way all the way back from your build process I'm sure compiling source into a binary on an untrusted developers machine that might have been compromised I'm the fact that you know we're all the source code came from doesn't really get you much awesome time for a demo so

we're gonna actually do two demos simultaneous we won involving physical patching in one involving computer patching we're gonna put the iron back an environment ready yeah erase a little bit okay yeah yes so I've got this github app here I set up it is running kubernetes and it is vulnerable to the shellshock vulnerability from 2014 anybody remembers that one I have an iron and an ironing board yeah so this is a modern application so it's got a docker file and a whole bunch of kubernetes yeah Mille I've got it running here inside of kubernetes there's a cloud native web page these are something called the cloud native gateway interface or CGI intake we did that in action here we

will click Submit my iron is plugged in so this applications running in kubernetes it just tells us a little bit of information about what it's running in and then which version of bash we're using this is a pretty old one I have a jean jacket that has wool in your arm so this is based off of an excellent github repo I found it makes checking if something is vulnerable to shell shock and exploiting it pretty trivial so this is the scrip we're going to use here and we're going to take a look here at this application I see my terminal I need iron-on strips awesome so we should've just defaced my web page yeah this is pure remote code execution so we

know we're affected by this now we're gonna go in and patch it pull up my editor open up my docker file and remove this line here that's installing a super old version of bash on purpose this was actually the hardest part to setup in this demo you guys anyone was doubting my iron is actually steaming I can hear it it's almost lean I don't know if this is even though I didn't really think through this patching phase okay let's figure it out okay I have a patch the kubernetes pad actually a sticker so it's probably not gonna stay on the on the jacket what we're gonna find out and Millennials who've never ironed before you're gonna want to put a cloth on top

of the Hat just everybody I know ironing I just built the new image I'm going to push it up through my registry hopefully the conference Wi-Fi works as well as the conference electricity yeah I'm awesome so now I'm going to come into my llamo file and change this to reference the new version of the code you're going to apply that I think this is on I'm gonna turn this off I think it's a tie yeah we're fixed and we're on a modern version of bash clap for the ironing so you know it's actually critical which is I have this patch jean jacket but you're wearing one that has a hole exactly so how are we gonna get that on you this

one's green this is critical in a second one arm at a time okay patched alright so you get the idea and now let's iron out the details would actually have miss Proctor so you might not have realized it but you just saw us through a Bluegreen deployment with physical jackets I took a few shortcuts in the demo but so if you haven't seen one of those before that I know we didn't explain it ahead of time a blueberry deployment is a way to manage your workloads so that you can roll out a change to production without affecting traffic I was able to get this jacket on before I took that one off completely so the way it works is you're

currently running a blue workload in production I mean you wanted to play a new green workload for the security team the new green workload is your patch version right now you already have new running taking all of your traffic and you find out if there's a new ode a vulnerability so you've attained that patch you roll out a parallel green deployment in production and then since you don't have a test environment you just send some traffic over to that gradually once your confidence working you move more and more traffic over and once everything is great you can kill the blue workload it's using blue rain deployments means that you can hatch security and production live as

members you means you can move oink into the container from one truck to another while they're both going at full speed it's like that terrible Stephen Matrix Reloaded so now this hasn't been specifically containers you can even do it with jackets so why do containers make this different they make deploying these changes more frequent and more straightforward which is how you're meant to do rollouts with containers so Google's infrastructure is containerized based on our board orchestration system was the predecessor for kubernetes and we use it to deploy services and security patches on upwards of 4 billion containers a week board runs a lot of key services at Google including Google compute engine and Google kubernetes

engine that's right we run containers and containers starting sometime in 2013 we realized we didn't want to have to deal with downtime for maintenance many of these maintenance events are disruptive they required us to reboot the hose machines our goal was to keep hardware and software updated across all of our data centers without having these restarts so we use something called live migration which is basically a Bluegreen deployment for running VMs when a new workload is deployed alongside the existing workload we use a load balancer to gradually move traffic over until it's fully handled by the new instance if you're confused how we can do this for running VMs remember again that VM is our that run inside of

containers in Borg there are three basic stages in this live migration and the first one the VM is still actually cuting on the source one most state is sent from the source to the target after that there's a very brief blackout period in the middle where we stop everything and move the primary execution point over to the target at the very end we have a post migration brownout period where the original source is still providing some services for the new target until migration is fully completed and specifically this one means that Google's infrastructure including Google Cloud were patched against her fleet in 2014 inspector in meltdown in 2018 with no downtime to our VMs and users not noticing Google was

also involved in the instant response for these only police so had pre-release access to the patches and even if you're not a large tech company with access to the patches ahead of time though containers mean that you can still start patching as soon as you know if you're even if you're pressed for time and avoid having to panic or take down at workloads in order to start securing them so altogether you know takeaways from today that means that with a switch to containerize microservices and a DevOps like frequent deployment model you can apply security changes more often you can enforce the software supply chain to control what's in your infrastructure and easily determine if you're affected by a new vulnerability

and with Bluegreen deployments like Google does with live migration you can patch your workloads live and yes container risk can be more secure they have the potential to do that since they approve your patching model containers hatched to learn more check out the blog post that Dan and I wrote that's the first link on exactly this topic and then there's also a link there to more information on container security and if we have time we'll look at the slider for questions no time so my last plug will just be if you're at RSA and born to come check out this space we're giving additional talks on container security and other security things [Applause]