Containers with Windows, Elevators and Backdoors

thanks everyone thanks for coming and kind of feel a little shy actually didn't want to do a big introduction but I do have a slide so all right so welcome to this talk and first I wanted to see you guys probably saw on the agenda we have two full hours of containers yay no well don't worry so Josh and I sink top ahead of time so we know who's gonna talk about what so you're gonna get really good material for the boat two hours if you wish to say for two hours so let me start with who I am and I'm currently a director of solutions engineering engineering at Eclipse him which does hardware and

firmware security I was also a did a stint as a security architect for check marks and people probably heard of checkmarks focusing on application security and then before that I was leading a group of professional services for Identity and Access Management so I've literally been up and down the security stack and over the years I got a lot of spirits and everything and just a lot of experience in being a hardened site of the stack so what the stock is not gonna be about I'm not here to talk about docker well actually I'm probably not kidding anyone I'm you can talk cannot not talk about docker I am gonna be talk about darker but just not just about docker so

what I'm also not gonna be talking about here in this presentation there's everything that's low on stack so I work in Hardware infirmary security and I work with those exploits but a majority of my experience is actually dealing with an enterprise level security as you've seen so it's everything up above the the hypervisor stack it's in the kernel seconds the user space stack everything else you see down below I hope everybody understands if you get owned by one of those things you screwed anyhow right your your really yeah there's no talk about continuous at that point so but I'm not going to talk about it I'm gonna talk about containers and when you talk about

containers in is if anybody's been into presentations about containers you know you have some obligatory slice so here's a slide about container history alright I'll just move on okay here's here's a slide about broken containers broken they're horrible okay here's a dead whale okay dead whale for you guys okay good now no that's out we can actually get to the presentation okay so I wanted to share his story and its story about me about a year ago so and this is where I've inherited what I call traditional some people call legacy I like to steal a call traditional security product it's a monolithic products it's focusing on enterprise level security it's it's focusing on certain things that developers do and

it's trying to analyze the work flow of the developers now developers are on the forefront of technology as I'm sure everybody have noticed sits they you know they're screaming containers they've been screaming for five years for six years they've been screaming microcell services for four years they now screaming serverless been screaming for two years there's all the cool and new technologists are coming on the market and they really want to try it they wanna you know make an impact build something cool build build something quick and they are my consumers I'm essentially working for them I need to provide a product that lives in their ecosystem and I'm stuck with the legacy product that lives in the ecosystem of

Windows you know dotnet development you know services Windows services and a sequel and all that stuff and I need to I have certain goals I need to make it secure I need to make it fast and I also need to make it so it fits inside of the DevOps infrastructure so essentially Dell ops is serving doubts and devs are making their crazy decisions and ops wants to put everything in the fast pipeline you want to put something in where you you can get a turnaround for their servers for the Dells and now I need to make that product the legacy product work in that pipeline and so so go okay well let's start with this

let's start with you know why you know containers are pretty cool I heard let's start looking at what what are they for and looking at it you realize that these are actually two goals and they're not necessarily competing goals there there's that's something that developers need and they're trying to close this gap and they're closing it in to kind of dissimilar ways but one and the more important one for the devs is distribution is how do i package it up in applications so it works smoothly in all the systems how would I you know put it in my dev box and then give it to the ops guy and then he can run it with no

changes so that's the distribution problem now the second problem is how to run untrusted code and by untrusted in in the dev sense it means somebody else's code it's like you know it's no js' I don't necessarily not trust it but it's no genius who knows where it can get out so it's better if I close it and that's the sandbox problem now they're not like I said mutually exclusive and this is where containers kind of come in play alright so back you know five seven years ago when you talked about container so you mentioned sandbox as the VMS came up right and we all been using VMs for a long long time vm is

actually pretty good about container raising applications right escaping from a hypervisor all to the systems you know not a thing for the past five six years is it possible probably is it likely very unlikely so they've been really good at keeping the close eye on you know everything that happens in the user space in the kernel space and then down to the hypervisor so they're actually pretty secure now the trouble with them is they're good for security they're not really good for distribution if anybody tried copying you know 10 50 and 50 gigabyte VM off to ops machine you know that right so there have been systems they've been solutions to try to make them a little more portable to kind of

like we sold the sandboxing problem now distribution maybe we can solve it with vagrants if anybody tried vagrant you basically your infrastructure as code you push it out and creates VMs it's good I use it I actually daily use it it manages the VMS alright but it's still kind of clunky and it still takes time and it's still not very sufficient same thing with sheriffs and puppets who try to manage the infrastructure at the code level - it works I mean it's a good workaround but it's really still not solving the distribution problem and this is where we come to the kernel level containers you know those things that kind of live in here and this is

where docker is right so the kernel containers are the things that isolate your security at the level of the kernel assist level calls and I I feel obligated to mention the the ways that are they're made if you actually want to find out more about the details of what goes into making a container go to the next talk josh is gonna talk more about those but I'll mention them here so when I say concur non-level containers are not first order primitives that means that they're not things in themselves there's sort of aggregation of other things it's like a container is a thing that's put together very cleverly and it's that there are other things that is

put together with are the capabilities which limit the process privileges the C groups which limited resource access then the namespaces that isolate kernel functions and then the second filters that filter the SIS calls into the kernel and finally you've got an overlay file system and an overlay network so take all those pieces by themselves they're not very much worth it but you know combine them into something new and you get something like rocket or docker and that's how containers came to be and so there are multiple ways the containers are done actually I'll start from the bottom you if anybody used Heroku you've already used containers you didn't even know that here okay we're the first ones

who actually created containers now they're not public containers that you can you know run hiroko containers in your box but they're the ones who actually came up with that concept and that's because they run untrusted code on their infrastructure but later after that and you can go back to the history slide if you really want it but lxc containers there's still a thing Alex see is what I call a hardened wood container it's like you know you're good all those primitives and you're the ones in charge and putting those together and you're the one who's enabling this and then once you've put it all together then you've got your car assembled and now you can actually run with it and

canonical is trying to help they're building something like lxd to manage those containers but it's still kind of like hardcore mode containers for you people use it especially people in big companies that need very fine-grained control over containers now the docker containers and this is where the whole thing took off right this is the easy mode containers there's the ones that you go and do you know docker run blah blah and that's suddenly poof you've got your system that's like we're cool and as you can imagine there's some cost to it we're gonna come to costs of the ease of use now there's rocket containers and I'm gonna talk a little bit about the security of rocket containers but they

they were created to be secure were like better River than darker okay and you know the other thing that I wanted to mention is when you talk about container level containers you actually do have really fine gain control control and you can run you can turn on and off features in those containers and do really cool stuff so for example I'll give you hopefully you can see this so here's a here's a container that runs a Microsoft by the way my I'm running Linux on here but here's a container that was gooey this the whole thing is inside of container if you not familiar this is sequel operation studio that Microsoft put out for managing sequel server and

you'll see my talk why I need to manage the sequel server but this is the Linux container with a gooey front to it and it's running it and if you actually want to look at it give it just second I'll put this so that's this guy you see I just forward X and I'll run the container and I get the X so that's your gooey container you're actually running stuff inside of the container but you're interacting it on the gooey level with it all right so the other thing really cool things is you can run obviously Windows tools will come to it and you can run a algorithm so back at home I actually have a little

lab where I'm running Nvidia CUDA deep dream Network and I create images with a deep deep dream Network and it's all running inside of a container so I didn't install anything I was like really really we were cool I got AI on my desktop with no like investment into building the crazy you know Python modules non Python and everything that goes with that and so I was really really interesting and the big point I want to make here is with all that flexibility comes responsibility so there's a lot of things you can shoot yourself in the foot with and we're coming to this but before we go into containers into kernel level containers I did want to mention the app level

containers and those are the things that you probably seen too so for example if if you ever tried if you run a gnome and you ever tried to install something from KDE like the Kate I headed err in KDE and I really like hidden you do the usual just you you do the usual app install Kate then I comes to you and says okay I'm gonna upgrade in 2-0 packages I'm gonna install a hundred and fifty two packages under your system and do you want to continue yes or no and the correct answer is No so obviously canonical and Debian everybody recognized that little problem that it brings to you and they tried working on it so snaps happens snaps

happened about five years ago and what what it is and if you watch Mark Shuttleworth presentations he's actually really good about describing them but they're he's describing them as the things that are here they're not containers in the sense that are isolated from your system they're containers in a sense that they're packaged up so essentially it's a packaged app application that has limited view into your system but you also get a full use of that system and so there's been a very solid change if you guys live on Ubuntu you'll notice that sooner than later if you before if you type in a name of the tool that is not on your system you'll get

prompt and on goal like this Camille not found use apt apt install for that program now if you do this on 1804 or 1800 1810 Lubuntu it comes up with this not for all programs but with this notice a little difference you see this pseudo snap they want you to run snap now so so what's happening behind it's I don't know if it's good or bad it's it's a thing now they basically they're pushing people to using snaps more and more and they actually have their own operating system I'll mention that then the balloons a core that's all about snaps but snaps is just one thing in the canonical space now and you can see it's basically thing

in the user space it's not extending out anywhere down into the kernel or the hypervisor level but there's things that existed for a long time and you've probably heard about this is an app and the thin app and Microsoft V apps now there are other packages like a Panna Jazeera installing flatback anybody who's lived in Linux probably knows those they you might not know but each tab in your chrome is actually container sandbox by itself and it's running inside of its own pit namespace meaning that the programs within that tab can't see anything else so it is a container it's like a sandbox inside of chrome now there are only containers in the sense that this is a container so that I

continue yeah but that's the app container so as I looked through all this different technologies in the past and as I looked at my challenge I it was pretty easy to realize the amps were too hard in distribution the app containers were to open so I was left with the middle I was left with the containers that are in the middle of space now I'm gonna give you a little lay down this is this is not me from a year ago this is me from today and this is like if I knew what's happening with containers this is what I've showed myself back here but this is what's happening with the containers now so to the very so right

here you see your VM containers right this is the application that's running on a guest kernel that's I isolated by the hypervisor from affecting anything on a host kernel so essentially your level of isolation is this hypervisor that's VM now the traditional kernel level container is like darker live in this space so you have an application you have a bunch of controls and syscall filters that prevents you from abusing your host and this filter is the one that's responsible for security of your containers and then you go down to the host kernel now the thing that happened probably a year ago maybe a little more is the Intel clear container so the Cata containers so what they looked at the

same problem and said okay VMs are too hard Cisco filters are kind of too flexible how do we combine the best and they came up with a mini OS special key SM kernel that this really fast booting kind of quick to execute KVM you know qmu combined into one package and simplified so you run this essentially it is a kind of VM container but it is also very transparent so you get the best best of both worlds in theory and then Google looked at same problem and Google said all right so how about we instead of doing that the hypervisor thing in the Cisco thing we actually create our own kernel and we do it as a shim kernel so

all your calls from application go to that shim kernel and then we filter it out and we kind of look like a kernel we're not actually a kernel but then we whatever we get from the horse kernel which sounds really really cool right it's like wow why why not do this and in it's all the cool ideas that the trouble with this is they're all introduced a lot of performance downgrades so somebody and I have a link at the bottom somebody did a performance analysis on all those kernels and the Google G visor came the worst it's like the slower than the freaking VM and then they looked at the Intel CC and I was like it was

painful to do those two and so when you actually look at the performance graph and I don't have it here because I didn't want to copy it from that web page but essentially says your docker things are the fastest your CCS are probably in the next then the GM's and then the Google G visor now that's could change in the future so in maybe at some point in time they're gonna you know own up and really build it onto something really cool and maybe that's the next thing in containers but so far it's it wasn't and obviously a year ago it wasn't a choice for me either so another thing that wasn't a choice for me is the scone and the scone

is the next cool thing that kind of happened so containers that are running in IntelliJ X and claves and that's another idea it's like yay how about we go one level down do you people know about us Jax I see I see so essentially it's a it's a completely independent encrypted area for code execution and for data manipulation that only the processor can access and you can switch the doubt and clave and then execute your trusted code on that so it really kind of pokes way from the application down to the hardware level and then brings it all up to the application level so that's like you can't get any more secure than that and my question to

you is did you foreshadow it yeah do you guys anybody see what I did there yeah all right for those who didn't get it for Shadow is an attack on those GX enclaves it just came out fairly recently and that's the thing that breaks stuff in the STX enclaves and plus they are horribly slow to you imagine bringing all the stuff down and then we encrypting and bring it all up that somebody estimated it's like actually the original paper that proposed these years and claves and containers coated that there were 18 times slower than a normal container so there you go III think it's an interesting technology to watch for but that's probably even farther out okay

so at this point a year ago I made my pick darker and other containers were it I needed to provide the flexibility I needed to provide the performance doctor was fitting I need to have reasonable security and I was planning to run trusted code I wasn't you know running our clients code on that so I I was fairly you know confident so that doctor will do it fairly well without resorting to the amps and the next thing that kind of hit me is was it was Windows so now I have to both windows on darker it was like okay well windows for darker was the thing already at the time for probably two three years

and I you know don't deep down into the space of windows containers trying to figure out what what the hell's going on there well I'll give you a quick gist I'm not gonna go into a lot of details but there used to be a Google darker toolbox which was running containers inside of the VirtualBox installed on your windows then Microsoft kind of owned up to the whole thing and they said all right so how about we just for Windows 10 which is probably devil machines we can use hyper-v to run your docker containers but for Windows Server we create a Windows Server container subsystem that actually does run containers so essentially they did provide access to the NT kernel features

from within your container so you don't have to have any any other stupid VM level in between because I'm sure you understand putting a VM in somewhere in the middle kind of negates the benefits of docker speed right so the trouble is hyper-v use exclusive istic that means you can't run VirtualBox when you're on hyper-v although I actually did beat it into existence and I can show I'm running hyper-v on the k vm in linux inside of the VirtualBox and it's it's horrible don't try it at home but if you ever feel masochistic I can't give you pointers so windows containers they run on Windows Server and Windows server core the images windows images have to

start with only two distributions like if you know docker you can start from scratch you can go and build your darker from the beginning for Windows you have to start either from the nano server which is their version of Windows server that can run anything but 64-bit and only dotnet core applications and if you don't know dotnet core is the lighter version of that it's more Linux friendly they're actually porting dotnet core to Linux and then the other thing is or you can base it on Windows server core which is an 11 gig based distribution and then you run it with dot net and you can have fooled at net and you can have 32-bit and of course

telemetry you know Micra so what you're doing you can turn it off but by default it's on and you don't even know it okay so that's how you run Windows on docker containers inside of Windows and then you can run Linux but that introduces a hyper-v VM there's no tricks to it you can't run Linux kernel on Windows kernel you have to have a VM and between that does something for you and that's their solution so I'm not sure you can ever expect Windows to run Linux containers natively but when you run it just know that there's a hyper-v VM underneath it and I'm going to talk about it in a second - so and then when

you run Windows containers on Linux you can actually do this and I'll show to you so just second hopefully you can hear me still so I'm gonna switch to another shell and this is this is my darker that's well I can't type so look at those two things so this is the client and the client is Linux and then this is the server and the server is Windows so if I do a darker image I have a list of images and you see whoo I can run Windows server core I can run another server I can even run Microsoft sequel server on my daugher so that's pretty cool notice the sizes - this is you know this

is a little different from Linux containers but it's all possible but the reason this possible is not a big trick really what what I'm doing actually in the background is I'm running this little thing the Ronix linux docker host which is a Windows server core VM which I'm forwarding the docker management interface out and and I'm connecting to the darker management interface and this is why I all talk about the darker management interface in just a second so come on complete for me yay all right so that's the trick so I'm feeding off a different system and running it here and my Linux environment okay so with that out what's the catch and the catch is simple it

still it works Microsoft's is still working on it they're bugs all over the place they're still on the implemented features did not working is buggy and I fortunately I had to spend like two days you know now debugging the network where it would die in the middle of a very intensive stream and it was horrible but I still have headaches about it but so another thing is nana servers headless you can't run any Google nana server windows server container it doesn't have an RDP built into it so you can't RDP into it so what that means is you can't have that nice GUI windows containers what I showed you from Linux just a second ago you can't have that on

Windows it's just not possible so maybe one day when Windows when Microsoft actually goes into implementing the RDP into Windows containers maybe then you can have your GUI from the windows containers and you also don't there's no way to update Windows containers and I go wait wait what what no no update on windows so yeah containers are immutable in the way that my croissants you to do it is they want you to re-upload vbase your image rebuild your image so every time you pull an image you actually are building on the latest fixpack that they build in so what it means to you if you're running Windows containers you have to rebuild your images that's the

only way you can get a Windows Update on the windows container alright moving on so let's talk a little bit about security of the windows containers because that's windows you have to talk about security and a couple of things one is apparently I have no idea why they do this but when darker porter their stuff two windows they rewrote it in dotnet instead of go and what it means to you is that now you can look at the source code you now you can reverse the source code and look at the go because you know there's dotnet just like Java you can you know look at the dotnet code reversed from the white level code so

one guy did it he opened it up he said all right cool let's look at the darker core DLL you know he found this little thing that passed packets from the docker control interface and he said let's se realized I wonder what I can do there with this so what he did is he built a DC realization attack on the docker dotnet DLL and he got himself to escalate the privileges from from the normal darker D into a darker user with net Authority system so that's the thing just making you aware that's not a horrible thing but it is a thing another thing that people did with Windows containers and this is really cool attack i I I will walk you really

quickly through this because I think it's cool it's worth knowing about it was demonstrated at blackhat 17 and it relies on several critical windows components basically what it is is somebody I developer goes out to on malicious website and a malicious website has JavaScript and that JavaScript wants to target your docker instance and by default just like you saw me here exposing that interface developers do expose the internal interfaces out to their systems and by default that will be bound to zero zero zero meaning to any interface so docker is listening on your machine on any interface for any action you can do with docker and what they do is that malicious JavaScript then reroutes back to the

darker daemon on your machine through a little tricks with DNS rebinding Microsoft has this LLVM thing for assigning DNS names but you can trick it is essentially saying you're going to pwned something and then that poll now resolves to your local Dhakal German darker daemon so the JavaScript from the browser is now talking to the dark on your machine and from that point on you have full ownage because you can actually create new containers they created a new containers they put a shell script into that container that container actually because you're running at boot privileges and I'll show them a second to that infected the mobi VM remember I talked about the VM running on Linux so in fact that made

persistence and so at that point in time every time you spawn off another container on your machine it was already rootkit it so if you're interested find out more about this they go into a lot of details but that's just it a kind of cool thing that you can do with the containers okay so another thing that I had to deal with is that I was working with very sensitive intellectual property I was dealing with encrypted installers because that's sensitive IP you don't want to get it out I had to have a sensitive deployment I had a special licensing protocol that I had to do basically I had to protect my employer at the time and I had to make

it secure now well so just to explain to you what's going on here with secrets and by the way there's not a conference talk going on about secrets at the same time but why do you show you something so let's say I'm building a container and I want to look at storing secret information and I build something like this I look at look at my darker file and it's something like I've got a secret here maybe I can store it my secret in the environment variable but if you look at darker you know that you shouldn't really storing this I can try to store it in the build arguments I can copy my super-secret file into and then

delete that super-secret file maybe that's good enough for me to build a docker container and actually you know get my secrets into the container without exposing them to the end-user well so I'm sure you guys heard about docker history if you haven't you should and the darker history is this so I should probably build this so I'm building it so I'm doing this and I'm gonna do this I'm gonna add a build because I'm smart I miss I've heard about security and I'm building a build arguments to assign my secret password at the build time so I can save it and not have to give it out to the customer so it builds and then I

do this I just want to check how it built I do a darker history I do my container secret come on complete for me I don't want to type all right and I see something really odd in this history so instead you know it kind of looks ugly but I build this little tool called darker historian and I do this and it gives me oh my gosh it gave me back my docker file and it gave me with the the secret here and well thankfully the file is deleted so shouldn't have any files in that but it also gave me this like key to secrets so all your secrets are exposed you can't save them in a docker

file well moreover actually take a look at this so here's my docker historian for let's see what do I have oh the sequel my my little Microsoft sequel file sequel o sequel Express ok so here's here's my secret my docker file it's essentially reversing this the docker file back into what it was so if you ever get an image and you want to find out how was build run this you're gonna get your darker file back with all the details moreover you know actually see this part is what darker did themselves this is how they built the image that I built my docker file from so you can reverse all the way back and just to show you I'm not I don't have

anything up my sleeve here's here's my docker file for Microsoft sequel so this is what I did and this is from docker so this is it reversed you see it looks a little bit different obviously it's not the same syntax but then this is what I got from so if I ever want to rebuild Ubuntu like they did and I can just run this file so that's easy alright so that that was about history now another thing is like all right what I do about the file and there's darker save and I do darker save and I save my secret to ah dump and let's see what's in that dump and I go start my my little

tool it stores right here okay I have the dump file and I look in the dump file sorry look in my dumb file and you know what's going on I okay so I look in the dump file and there's several layers in that dump file and there's a JSON in that dump file that actually uh interesting to you can look at all the details of the JSON file and see what's going on there you can see the secrets too if you want it to but what's more interesting is that docker file has layers and in one on the layers I find the super secret file and this is because that's what docker does it never

deletes stuff it actually creates layers yeah the file system is right on or create on right so whatever you write something it creates another layer so your secret super secret file is still there don't worry about this okay so that's about secrets and that gave me a huge headache I had to look at it in its like oh my gosh I was building darker and I built this well turns out they're solutions and the solutions are either squashing or multistage built I'm not going to go into details unfortunately that's not supported on Windows so I wasn't nowhere you know pretty bad move now there are open source solutions like docket of old and others like hash warp

and kubernetes and swarm that wasn't overkill for me at the time so what I did is I build a standalone docker container to contain all the secrets so during a build time they can get the secrets from the standalone docker container but that's essentially the extent you have to go to to actually build a secure and reasonable solution and then last thing I have to do is okay now I have to have a standalone Linux container so let's look at the Linux security and the Linux security has gotten a lot of attention a Linux acha contain container security got a lot of attention over the past years so I'm not gonna go again into details if you're really interested in

figuring out if they're secure the short answer yeah layer secure no talk goes about referencing just Raziel and her container AF thing basically it's proving that they are secure unless you do stupid stuff and I appreciate people do stupid stuff but there's you can do protections against the DOS you can there's cap net raw capability in the Linux containers that in theory give you men in the middle but in practice nobody actually proved it there's also a knowledge set of tools that allows you to look into the application side of your container and and defend from the application level your container but there are tons of food guns and food guns is something you know you pretty

much do to yourself and this is how you screw yourself over so here's a typical scenario if you ever run know the inside of a docker container you probably encounter this when you hit control-c the container doesn't terminate and you do what you Google right and the first thing that pops up is this guy from darker thread saying okay if you can't terminate it just add the - - bid equals cost and and you're good with it no change to the code and you're like yay well what this does is it exposes all the processes from your host to your dock in containers so whatever is running inside of your container can now you look at the command lines on your

poor host and it's like yay so the actual answer is to add single handle inside of your app there are other food gowns that people do to themselves like the rude thing so let me show you the rude thing so never run containers root you might have heard this but here's to use the actual proof so docker run and do interactive with the volume of slash mounted to slash are on your container and do all porn because it's small and run this so this is darker container and notice I'm running it as myself right it I'm user I'm not doing soo do anything but then I'm kicking off this container and I do inside of that

container I do crap for roots inside of /r remember that's where I mounted in so I basically got a root on my machine without being rude and the reason it works is because the Dr Dee container is running as root and I'm a part of the doctor group so that last group that essentially gives any container Iran root privileges so if you run root inside of you container you root on your host that's it there's no joking about it so don't just don't okay the other thing that people do and I actually did it in front of you guys if you haven't noticed its supposing the socket so socket is the way the doctor D communicates to

yourself so when I showed you my little trick with Windows that's the socket in Windows case this is a TCP socket in the Linux case it's the actual socket so when I'm exposing my socket I'm exposing myself to that attack I just showed you earlier and that's not just Windows Linux - if I can write to the socket Windows doctor doesn't do any authentication it'll just trust it it's like you can you can talk to me alright I understand I'll run the container for you and if you you're running it as root 2 so if you if you do something try not to do this darker sock thing exposure sometimes you have to do it and if you

need to do talk to me you know how to work around it but yeah windows is kind of like that just to show you again why not to expose your darker socket in 2018 the darker had had 18 back toward images the remaining Manero and it's a known thing you know docker kind of warmed up to it and they killed the this