Containers for Pentesters by Rory McCune

welcome along to this talk on containers for pen testers so I'm guessing by now a lot of people and we've even seen examples in the previous talk um of people using containers for a variety of things um what I wanted to do with this talk was talk a bit about containers I'm gonna have to go where it wants to go uh and then talk a bit about how they work because when we are looking to use things especially as pen testers which is kind of the point of this talk we want to understand how they work because if you're doing pen testing you can do some funny stuff it's easier to do that if you understand what your tools are doing and how they actually operate and then I'll talk about some gotchas and some ideas you can use when you actually want to make use of these things so before I get started on the talk uh oh that's not gonna work let's do this very quick about me why am I giving this talk so uh my background is pen testing uh I'm told fairly recently I was a pen tester I did this for a number of companies for a number of years uh laterally I was looking at a lot of containerized environments love Docker a lot kubernetes uh these days I'm a senior security Advocate at datadoc if you've not heard of datadog we're a large SAS observability and security company um a couple of things I do in the kind of cloud native community of kubernetes container Community I helped maintain the CIS benchmarks for Docker and kubernetes has anyone ever used either of those Ci's benchmarks or uci's benchmarks okay if you haven't if people haven't ucis benchmarks they are vendor neutral um hardening guides that you can get for pretty much any technology you want I helped maintain the ones for Docker and kubernetes I'm also a member of kubernetes six security and the cncf tag security these are kind of like special interest groups so if you're interested in like Cloud native security or container security um they're very easy to get involved with you just come along to a zoom call either weekly or bi-weekly and they're a good place to find out what's going on to get some new discussions so if this is an area that you find yourself interested in very much recommend coming along to those so the first thing I wanted to talk about was what is a container because this is really important and not necessarily that widely understood um essentially when you start a Linux container what you are doing is starting a process right you're starting a program just like you're starting any other program so containers are just processes it's an important thing to remember the way that containers differ from your ordinary processes is all these kind of boxes that go around it and these are all different ways that essentially the process is isolated from any other process running on that machine or isolated from the underlying host that's why if you're in a container you can't see by default all the files from the rest of the host these are all existing Linux features these are all things Linux does you can use these without using containers and I'm not going to go a lot in depth into those because it's quite a long topic if that one is one that's of Interest like how exactly do namespaces work then there's actually a series of blogs that I'm doing at the moment on desktop security Labs where we go into a lot of detail about each of those at one in turn and I've got a link to that at the end of the uh at the end of the talk now as the last speaker was willing to do live demos I thought why not let's try and do some live demos so I'm going to demonstrate this try and make it a bit more real a bit more kind of like yep that's what's going on uh so let's go to right I've got a machine here this is uh just a Linux host running docker and I want to do PS uh we're gonna do PS ax FC engine X so what I'm doing here is I'm saying to this machine do you have any nginx web servers running the answer is no I don't Okay cool so then we'll do is we'll say docker run helps back in time not that one that one we're going to use Docker run and we're going to run a container based on the nginx image which is that bit at the end we're going to get a name we'll call it web server and we'll write in the background so at this point I'm now running a Docker container that's what I just did I rocket container however from the machines perspective from this host perspective what have I done I've started nginx as far as that host is concerned all I did was start the nginx process running on the machine just as though I'd installed nginx and started running it because it doesn't know about containers underlying host doesn't know what container is it just knows about processes so it thinks ah you're running a process then and I'm going to make note of this PID this process ID because one of the cool things that once you know that containers are just processes you can interact with them just by using Linux process tools you don't have to use the container tools and we can we can demonstrate that let's put this on the top we'll do Docker exec I'm gonna do Docker exec so Docker exec just executes a command inside a container and I'm going to create a new file by saying touch my new file the main thing I want right I'm just going to create a file inside the container once I've done that and then say I want to get access to that file say I want to mess around with the files on my container and I don't want to use the standard container tools well because we know that containers are just processes I can go and find that processes file system and play with it and in this case all we need to do is use sudo LS slash proc and then my process ID root what I'm doing here is I'm going to list the files in a file in a directory in the proc file system in Linux now the proc file system is a special file system in Linux it contains information about every process running on the machine amongst a lot of other things and one of the things you can do in the prop file system is get access to the root file system of any process containers are just processes I can get access to any container file system by going through prop and as we can see I can so you see there all those files there in the root file system are essentially what comes from the nginx image plus my new file which is the file I just created so literally once you know containers are processors you can just play with use them like processes and if you're a pen tester you're thinking well how can I use these things well hopefully that makes a bit easier if you think okay I understand how this works so demo one worked awesome so what's Docker right so I use the docker command there the stuff I've been running so far has been Docker commands there are other container tools available but Docker is definitely the most used one if you're getting involved with containers for the first time I would recommend starting with Docker just because there's lots of tutorials and other information and it's easy to get hold of what does Docker do well look is actually fairly straightforward Docker is a client so the docker client which I was just running there is a goal line binary client you can move that file around to different machines work the same way it talks to the docker demon and it talks over a Unix socket file so Unix socket file is just a way of making a server available without putting on the network but basically it's talking to a web server the docker Daemon is just a rest API if you're used to using rest apis you usually web app stuff then Docker demons pretty much a rest API the docker demon and then we'll go and get any images it needs from a container registry it could be Docker Hub could be any of the other many many Registries which is also a rest API so if you're a pen tester and you've got a background in web app testing or web API testing you'll find that a lot of the stuff in container land is just a series of rest apis talking to each other they just hide it really well and there's like some nice fancy guis um and clis and then all of these are starting containers so that's all Docker actually does it basically says gives a rest API command to Docker Damon which then starts a process on the machine so it's actually not too complicated and we cannot demonstrate a bit about how it works so if I do so count so if I do this command socat if you've not come across it is a really handy tool for playing with traffic and intersecting things this socat command basically just says give me a new socket file and then send any traffic you get to that socket file to the docker socket my so what I'm going to do is actually is put something as an intersection so I can look at traffic and see what's going by if I hit enter on that it will sit there quite happily and uh and listen and then in the other terminal if I do see if I then tell this terminal hey Docker please go and talk to my temporary socket file that I just created and give me a list of container images he'll do that in this terminal it says okay there's your list of images what I would expect but back here we can actually see what happened and literally it's just a rest API what it does it starts off sending a head message to underscore ping so this is hey Docker Damon are you alive are you there it says yes I am I've got very nice distinctive user agent if you're ever looking for it gets back at 200. and it says great go to this API endpoint so just API version images Json really simple rest API and Docker comes back and says cool there is a list of all the things big Json blog so doc really that's it what it does it's all it does it's really fairly simple it says I'm going to tell you to do various things with containers and you're going to launch them up for me one thing to mention um as pen testers you probably sometimes you sometimes need like fine grain control over what your machine what your tools are doing like what exact networking you're doing you do any low level testing anything to do with like Network sniffing Docker desktop is what you will use out of the box if you are running on Windows or Mac um Docker desktop hides some complexity from you it looks very nice and it works really well but if you're doing networking I would kind of recommend avoiding it because what it does is it inserts this Docker desktop thing in the middle and it actually hides a Docker VM and that virtual machine you can't even see if you use like if you're on Windows you can't even see it in hyper-v manager it will not show up and it works really nicely if you're just doing developer work or you're doing like high level work if you're doing any network testing I would recommend avoiding this because it does quite a lot of fancy Network magic is the only way I can put it and it's just going to get in the way of your tools what I would typically recommend is getting a VM you manage yourself and installing Docker on top of it don't use Docker desktop just worth knowing as pen testers really nice piece of software but does a lot of magic I don't like magic when I'm pen testing because it things break in the middle of a test I'm like why did you break and the answer is some complex magic when broke on me so a very important thing to know about Docker is Docker security model with the security model is what I always describe as flexible um what it is is Docker will say that anyone who can run Docker commands on machine can be root on the machine that is by Design that's not Elite hack it's just the way it works it has all these different layers of isolation that I showed you on the graph earlier on and it says anyone who can run Docker commands can remove all of those layers of isolation at any time in any way they want so the culmination of this is this command here this rather long looking command this was from a Blog in 2015 by a guy who called it the most pointless Docker command ever when I was pen testing this was my favorite Docker command and I know how sad it is to have a favorite document command this is my favorite Docker command and I'll and I'll show you why because we can demonstrate this one super simply Focus so this Docker command very long what it basically does is this it first runs a flag called privileged and what actually has to talk to the guy in Docker who designed this flag and he said they wanted to call it insecure but their management wouldn't let them but that's what it is that's the removal the security flag which is great that it gives you one of those is the being of continual security existence the fact this thing was ever made because it's basically the total security off but it makes things work right so a lot of people will do it because hey things work really easily if I turn on security off we then say okay you know how we were getting an isolated file systems and networking everything we don't want any of that we want the hosts networking we want the hosts process list we want to host everything we then want to mount the root file system from our local machine into the container and then we want to run the command host and you can guess what happens when I run this before I run this command I'm an ordinary user on this machine when I run this command I'm the root user it's that simple is that straightforward that is by Design that's how Docker works if you let anyone run containers on a machine if you're a pen tester and you find a Docker demon just run that command and you get to be rude if you're doing kubernetes and you've got lots and lots of machines which basically kubernetes just runs lots of Docker machines on different hosts you can run kubernetes commands that do the exact same thing and you get to be root on every single machine in the cluster this is by design of the box that's how containers are meant to work it's very important to know when you're using them that's just how they work so why do we need these things as pen testers this was kind of the point of the talk um we've seen how they work we've seen what they do why is pen testers would we want to use these things for my money there's a couple of reasons um first one is um if you're a pen tester you have a new environment for every customer every week and of course you never ever keep an environment from previous times that might leave test data from previous customers in the same place that has never happened to any pen tester ever yeah um containers make it a bit easier to keep that stuff clean right because whenever you start a container you get a new ephemeral environment right it goes away as soon as the all the configuration goes as soon as you stop that container you start another one it starts cleaning from the same template you had before containers make it easy to get clean environments for your tools so you don't run that risk that I'm going to be all you know hanging dates with me hanging from previous times another thing is I'm guessing anyone who's ever been a pen test or has found some tool there aren't some obscure technology some web app that they don't normally do and they find a tool that someone wrote seven or eight years ago using a really old rookie version of pearl or python or node or whatever and it uses a whole lot of old libraries and old versions of language the last thing you want to be doing is installing all of those on your shared virtual machine because if you do that long enough you will break your machine and everything will go wrong and node or python will start complaining horribly containers let you isolate those things in a nice little bundle that stays isolated and away from anything else so you don't have to worry about the fact it's going to make a right mess of my machine this is probably one of my favorite places to use containers is you can run old software all the way back to you know really old versions of browsers whatever else in a way that won't make a mess of your virtual machines so containers are good for that nothing they're good for is they're very easy to maintain we've been seeing some talks about CI CD and GitHub actions and stuff like that it's really easy to make maintain containers using cicd you can have I'll show I usually have mine the ones I have I basically ping them every week they refresh and they rebuild so I don't want to pull my container it's no more than a week old every single time and again some customers were picky about you rocking up with really old out-of-day software so you can say hey look my software is already new because it gets maintained automatically last time is obviously as pen testers you're going to ask to test things like kubernetes clusters and the customer's defaults place will be give me your container image you want to run so you're going to have to have them because customers expect so VM versus container obviously lots of people most testers I would say probably use Virtual machines I'm not going to tell you that it's always right to use containers sometimes virtual machines are the best option um the main difference for my money is size a VM image will end up being 10 20 30 40 gig quite easily containers you can get them all the way down to I think the smallest viable containers measured in kilobytes but realistically even my biggest most bloated container image that I just Chuck everything into is one or two gig so it's much easier to move around to say to a customer here is this thing it's much smaller you don't have to worry about like you know file system limits or stuff all that as much but ultimately there's different ways of looking at it so see if you say you want to use containers and you're thinking to yourself where am I going to get my container images from the first place you will think to go to is dockerham Docker Hub has somewhere north of 8 million images currently available for you to download and run of those 8 million about 200 ish are what I would call semi-trusted as in they're maintained by Docker and either a commercial company or an open source project actually says we maintain them the other whatever 8 million minus a couple of hundred is are totally untrusted could be literally anything doesn't matter what they say they are there is no curation at all they might take down malware where they get notified of it but realistically speaking most of the time that just means you'll get stuff that hasn't been patched in five years six years now there's this stuff up there just never gets passed the person puts the image up the user web they use it for they forget about it if you were searching for it you might end up pulling it down you get outdated software but you could also get images that are called like I think there are type of squatting attacks there is active malware on Docker Hub there is anything you like typically the best thing to do is I would say that the official images never pull directed from Docker for production tests you're doing some research on a throwaway laptop fine don't worry too much about it if you're going to a customer's production environment and you're going to start running the tools from these images against customers production do not do this from Docker hub the reason I mention this specifically is this these are myself my images on Docker hub um these and these images the top one there is literally just Alpine Linux with a couple of networking tools that I literally did for throwaway years ago the next two are my kind of like kitchen sink where I put all my container tools and those are the download stats I can promise you that I'm not even 0.1 of a percent of those download stats so some people somewhere in the world have decided to make heavy use of these images that I mean I put up there I make no profits about maintaining them not putting the malware in them not rickrolling anyone who runs them I see nothing about them they're known as any Assurance but that does not stop people doing it laptop is 1.75 million downloads I don't know who's doing