Build To Hack, Hack To Build

yeah welcome to my talk Tartabull to hack hack to build or hacked Bulbul to hack I confused the two quite often so yeah Who am I my name is Chris Leroy I'm a skirt engineer at Heroku you can find me on the internet at Bron pony if you're on the back story between that hand behind that handle feel free to throw something at me and ask and yeah I like hacking stuff and I'm particularly lazy in such a way that I like to build stuff that makes me hack stuff a little bit quicker so both a couple things to hack we're windows android and now containers so this slides more for me than it is for

you so I don't forget what it is over to share with you today of course there was a problem we look at some existing research then I'm gonna introduce Bob with the silent T then we look at some capabilities and integration and of course a conclusion so that's just a rough outline so what are the problems or problem so I work in a very modern cloud DevOps hdl-c agile environment no machine learning yet but one of the issues that are often ran into was how do we identify and exploit container vulnerabilities which is quite a common problem with a European test or an engineer but after dealing with engineers quite a lot I ran into that

last point which is kind of the same problem but slightly different and once you're popping containers and identifying vulnerabilities in containers in a large-scale environments how do you test secure and monitor and when you're doing rapid deployment using thousands of deployments a day with a lot of containers and a lot of things that can go wrong it becomes a little bit more interesting and a little bit more than just popping shells and getting grid on a box so with that problem in hand and particularly looking at popping containers this is some existing categories of tools out there that I found quite useful this is a partial list this is by no means definitely not all the tools available

but when you're under container one of the first things you do especially for pen testers you run amuk contained because depending on the syscalls and capabilities that are available in that container that will dictate what you can do in that container and my content will tell you that so one of the first things you're going to do when you want to break out of a container then there's Google container tools one of my favorite tools at the moment is the container structure test really great way to statically analyze an image and work with the tile layers you can also work with the container as well but there's a whole bunch of other tools in that Reaper that are actually really

useful then there's clear which is one of the more older older but more it's been around longer than a lot of others in terms of vulnerability analysis and containers so static analysis containers your container may have vulnerability X in it then there's equal security who's actually released a lot of other tools as well but there's this one particular category so there's the docker benchmark which is quite useful in an environment when you want to benchmark your containers from a security perspective and then it actually is quite useful so these were the categories of tools that I found existed that I could use from a pen testing background and from an engineering background and this research

deserves its own slide and this was actually said besides Barcelona and this was released by a researcher called Alena Radu I apologize if I'm pronouncing their name wrong please correct me but this is really cool because while I was doing this research I saw this talk and this was one of the first pieces of heavily directed towards CRC Department technology and containers so it's not something that you if you're in a container it's not the binary that you just drop on the host but it is a framework targeted towards your CRC the environment and it's very very cool in terms that so I really recommend having a look at that and I saw this now so

well no this was really great if you have an existing hdl-c and see a CD that can support this framework so it can be done so how do I like to solve problems so I'm run engineer I'm not a dev I just break I break stuff sorry I'm social courted social probably curse less but I like to break stuff and to do this I created something called break after box or Bob with the silent e I have very bad reputation for tool names so that may change in the future but anyway written in go because I just wanted to embrace the in the hipster in East London and it's released as a binary so it's not a framework it's a

binary that if you're on a host you want to execute like I said I'm hot lazy so I mean I'm in a host I want to auto pound stuff of course I want to understand what it is that I'm running I wrote something that's going to allow me to auto pound common container vulnerabilities because after you get the ten shell on a container you kind of get bored of running the same commands over and over it also helps me to perform common container recon functions so thing Linux post exploitation but there are a lot of quirks in containers that you'll find along the way and I'll discuss some of those and also in terms of these capabilities I implemented them

so that you can perform them in a way that's useful for engineers and pain testers so not about popping shells of course it's there for that but also in a way that you can use it with engineers and in clcd environments and I'll show you how I actually went about that and I did make the github repo publicly available this morning you can find it there remember Bob with the silent T but T is in the URL and there's a lot of examples and all over there so first thing we're gonna do let's just burn all the things and if your opponent on containers Dakka Dakka Dakka dot suck is the first thing you're going to do what

is dr. it's a UNIX domain socket using UNIX systems for inter-process communication it is pretty much the core thing when it comes to running docker how you're going to communicate it runs those routes and if anything is part of the docker group you can essentially become routes on the host so why is this important well when you're hacking containers this is like the hollow wall to hacking containers it's not new and this is one of the first things that you look at so where this came from is that there are a lot of instances where people wanted to use docker within the container I'm going you know getting really inception like don't do this by

the way there are better ways to do that but that's where this came out and that's going to allow you to do two things it can allow you to break out of a container onto the host also it can give you a local provost so you'll find in a lot of situations based in CRC D environments you'll be given your own ec2 instance or your own Google Cloud compute instance and dr. dot sock will be there because sometimes you run within that environment but by default you'll be running as a non root user and you know if you're not root you don't have the keys to the kingdom you can use this to break out and become root and

that's has some interesting consequences actually quite recently so hydroponic sock so firstly you gotta identify polka dot sock you got to interact with it so depending if it's mounted on the interface on a UNIX domain soccer do you need something that can speak specific data grams for that so either curl or the doctor client there's a few other ways you can do that I want to interact with this it's basically the API or token you can say create a new content or mount this interface everything that you can do with the doctor clients is done by the sockets what you then want to just create a new container and you want to mount the hosts container host file

system in the container and then you then want to run in you want to run into this container and then you want to see eight root containers and the CH util is going to come around sometime in this presentation but you want to see 8 root the amount that you didn't step three in your new host and then basically it is like your root on the host with access to everything really simple really easy and a lot of fun as you can see there's a wash rinse repeat method over here so there are some quirks when it comes to this kind of stuff in containers looking for sockets can get tricky because living off the land in containers it's a

lot of fun but it might be the case that the useful commands that you need to find sockets might not be available so for example if you run good old docker image it's very limited and there's not a lot of commands and that's not just learn you contain the image that it's like that so find or SS and they said or whatever might not be in the container so if you don't need these commands literally one-line a batch that you could normally do to find sockets are not there another issue that you have with this is that the cock-eyed might not be mounted at wall run docker socket might be mansur that moobot so now if

you gotta traverse this entire file system and look for sockets and if you don't have the commands that you normally used this can get a little tricky so how do we solve this well such calls and this is where Emma contain comes into play just because the the command is or the binary isn't available doesn't mean that the circles are required to do that aren't available so on left those are the scores for a standard Ubuntu system that are required for the find command on the right is some go code to basically implement that so the find command might not be on the ear installed in the container but the syscalls might be available and the

really nice thing about go yes that is very terrible go code like I said I'm not add if I break stuff but you'll see that there's no shell come on it's being executed and the great thing about go is that'll go and execute underlying Siskel's so they didn't have to rely on shell commands which is really neat so same same but different so how do we actually go ahead and do this so show controls and so what we have over here so I'm on my host on my boon to host I'm just gonna watch a directory in temp in the top we're gonna run into a container and you can see we've mounted a socket at temp this is not a

socket and we're just gonna mount a local directory that has the binary in it so once we go to Bob we're gonna tell Bob always run Bob first his check it actually can run in your system and we're gonna tell Bob tell me what you can do so there's a whole bunch of stuff but first things that we're gonna do is we're going to look for sockets so if you look for UNIX domain sockets because there are other kinds of things you can do and in this case we find the socket at forth slash temp this is not a socket what you do what do we do now we google furiously on how deep own sockets now we

say Auto current equals true and now what Bob's going to go ahead and do it's now identified the socket and now it's gonna check if it is a talker socket and if it is it's now going to give you a shell in a container which gives you a shell on the underlying host so now we are on the underlying hosts file system and we're gonna go into the Bob folder and we're gonna create a file and at the bottom terminal you'll see that the father I'm going to create is now going to appear on the host operating system so once that decides to come there you go so now the host operating system we've now played with the operating

system and then you can exit with Bob you give an interactive TTY and now we're back on the container that we were in if you go back to temp we won't see what we were there so that is the hollow world - breaking out of containers has been around for ages you'd be surprised how common is there's a lot of security through obscurity but hey you know people going to look for voir undoctored suck let's man sit at bla bla bla bla bla because they'll never find it well you can so in this case Bob can be used to get a show on the host I think pentester you know nothing you know it says better

in a report saying uh we got rid on the underlying host and we could read you know at see shadow sure engineers how can they users they don't necessarily want to pop shells so return codes are very interesting so in your see our CD environments Michelle doesn't really mean much but you want something I have to say hey this is exploitable this isn't what will make use of that and I'll show you that next and then of course the last point um when you run the binaries Bob it's got all the packages or utilities that you need um if the schools aren't available that needs to do something you'll get an error I did leave a note to insert a

Linux capabilities joke but it wasn't there without being really cheesy I think capsules make joke wasn't available awkward silence okay that's cool I had to do that at least once so how does this look like in an engineering environment so so we're going to do the same thing but here we're using um Heroku SIA so normal CR technology and we've now done a git push and the very informative aramid commit message and now in our CI environment we said R and Bob author PO but CSC D equals true so what that means is that we're not going to run into drop into a TTY we're gonna tell Bob to set a exit code now if

you're familiar with CSEE technologies and testing frameworks if the exit codes are more than zero it means something has happened if it's zero you get the thumbs-up and while this is running so we'll get to the interesting stuff so you can see here bob has run in this environment now our staging environments and IC see our CD container environment is the same as production where you are using those assumptions so whatever we're testing in staging should be the same as production send as you can see over here Bob found some sockets but they were not able to exploit them because naca sockets so the test will pass because we're not mounting dr. dot sock within our containers and this is pretty

much how Bob will work in the CI environments the X occurred was zero the test passed so in terms of dr. dot sock if you don't need dr. dot sock in a container don't man sense there are better ways to do it via Dan and a few other technologies but if you do need to authentication is your friend there are a few technologies that do this so there are certificates and of course if you are using UNIX domain sockets there is inter process communication authentication that can be done that is specific to UNIX domain sockets really cool and have a look at that but there's a whole bunch of hardening resources there to do that

but there's more to containers and just burning dr. sock there's also environment variables so I think Linux post exploitation but especially in containers so you've got good old in VN path there you know environment variables just happens to be the secrets management approach for a lot of container technologies and there's also proc FS is definitely your friend why because just because you may have cleared your environment doesn't mean that the environment has been cared for specific processes on the system go ahead and do this look at proc whatever PID and environ you'll sometimes see that there's stuff there that is not in your standard shell environments so it's very good to look at that often some

juicy stuff there I'm so yeah Bob accesses both so basically OS environment will give you the key value pair and then also to access proc and all the pids in the system now this can get a little bit tricky containers why well because ephemerality is that's a sigh frustration there so by the time you've list you've listed the contents of proc and then go ahead an attempted to read prop ID in barn that process might not exist anymore so there's a saying about when something hits the fan generally happens over there and then of course there's just proc FS which is just you know a virtual file systems just can make your life hell I definitely have some hair loss

due to that so I'm wearing the hat and that's pretty much in go code you can see that there's very little shell commands Miami executing actually I shout c'mon is bit of a hack there too very far the contents of proc PID because it can get a bit tricky that the file exists there is actually a file descriptor for proc one environment there's actually no contents in it so that's where some of the weird magic comes in those proc ESS so but we can go ahead and actually analyze the stuff so how do we actually analyze it show controls make it fullscreen so we're going to run a docker container and we're gonna add an

environment variable so touch of approval super secret password and we're gonna run Bob so Bob by default so if you use the Recon very hacker super cool option there it'll go look in proc and you'll see it's found a whole bunch of processes that have that environments attached to it um so it's your shell at hand that you actually know your bash environments and I'll go ahead and actually look at envy as well because stuff might be hiding in butts and by default Bob will look for secret and password you can also go ahead and specify a word list you know it's 2019 and word lists are super cool and in this case we're gonna

tell Bob to look for once I've read Jack points cool yes we're looking for yellow secret find me and Bob and we're gonna remove that and we're gonna tell Bob you know just look for some word that will come up pass christos something what it has find me cool so yeah so we're gonna look for find me in environment variables and we should get nothing and there we go and if you look at the return code that will come out actually be 0 because Bob returned nothing in the previous test when we had all that output for super super password the return code would have been 1 which is what I'm going to show you now and there you go so that's

super secret password being found and the return code is one that's been a bit cut off by that return code is 1 or 2 so there you can see from a pen test environment you're on on a host you want to scrape that information you can do it an engineer you can drop it in your CS CD environment which is what we're going to do over here so inside our CI CD test we've said run a script and I'll show you what that script looks like in a bit but we're going to run two tests so we're going to call a script and that script has a test for looking for owning UNIX sockets and then there's a second

test that's going to use a word list and search for any of those key terms inside our OS and viral or in our proc envira and that's going to be run on every pool it or every pool request or every push on our repo so I'm gonna fast forward a bit because I have definitely made great too much content for today and I probably won't get it in 30 minutes but what we'll see now is one that once the test is running well we'll get and there we go so the test has run so the socket stuff that we ran previously that was the first test and our Bob has gone ahead of than any environmental variable

test from the words that we used and nothing in protein bar or in OS that E&V contains those key words so we do we're getting green messages all across so in terms of environment variables you'll see there that Bob didn't need the C ICD flag because there's no TTY being returned so Bob by default will set return codes on the flag that you're setting so you don't have to say that CRC B equals two like I mentioned you can supply word list or by default or look for password and secrets and in terms of remediation well this is a tricky one sequence management is hard and it's mostly Turtles at the end of the day if you're going to use in v4

secret stuff threat model maybe so you can know what you're dealing with and sometimes it is quite useful to actually nonce your data but be careful with it because if you actually threaten all that you'll find that you'll be creating race conditions for yourself so it's an interesting problem sequence management in the cloud but yeah there's more time to get matter um but it was a bit of a millennial reference but that's alright pads containers love living in class who would have known and because of this you can find your metadata services in containers so good old one six nine two five four it's in AWS it's in G CP as soon as you it's a whole bunch of other class and

this is really cool because networking is hard in containers and when these endpoints and metadata services are exposed depending on the platform you can do a lot of cool so for example net instance say that kokum and if you are in a easy to instance and if that specific that specific instance has an R and I am policy or role associated to that instance you can access the credentials through the metadata service now there's a whole bunch of other stuff you can do depending on the platform for mounting volumes to creating hosts it depends on the platform but it's definitely something that you want to look out for but that's not the only kind of endpoint inside containers

there's girl API so think control plane so you kubernetes control plane journey running on 10001 or whatnot or any other endpoints there's stuff to be had over there and you never really know what's lurking because a lot of time containers just get access to it because you know routing is hard and IP tables you know IP tables in a container hmm it depends how much you hate your life that you want to do that then of course there's got all easy to classic and if you take a quote out of the documentation in ec2 classic ec2 instances run in a single flat network that you share with other customers should you not it is 2019 and

that is still a thing so depending on what you're doing either what you're mounting on interfaces is exposed to everyone else and vice versa so you kind of want to check that and of course you could go back to container quirks the come on that you want to do so you know you're on the box so you want to in map you know what you want to do RP address show or I have config now tell me what I'm working with sometimes you don't have that inner container so it gets a little bit hot so of course let's have some schools that will do that for us so really simple example with Bob is this

is a feature that I want to you know definitely build out a lot more but Bob by default will look for two to two metadata services and we just say metadata equals true so if you don't provide an endpoint list Bob will look for 16.25 for kubernetes not default at SVC and you see there we got nothing so now we're going to say you know what see if you can hit Heroku from the container and we're gonna provide an in-point list and what Bob does and if it gets a valid HTTP response from any of those endpoints it will change the return code and tell you that okay got a response from Heroku calm and the response was a 200 now if

you see the response code has changed and that can be quite useful in environments where you want to actually go ahead and test with these kinds of things so like I mentioned these endpoints return codes too far for a default and in terms of remediation the stuff isn't that hard it's just you know your access controllers but understand what is their taste to environments for metadata services because it's I don't think it's that obvious that when you building stuff in the cloud that these things are available and then of course you know routing is a friend and authentication like we've been securing endpoints for quite a while but if you don't if your endpoints are there you

don't know what you've got to secure so all of this we can't stuff is fun and there's a few other recon options are getting interfaces IP addresses port scanning all the fun post exploit stuff but fish with containers there's binary hijacking and this is if you really want to break some so so often there is the case that container binaries are executed by outside entities for example dock exec if a containers running and you want to enlist the contents in a container you say docket exec Baca IDE LS and you get the contents of that container cool all right trust the command at all like it does not check am I actually running LS in the container

sure if it's your container you know you're pounding yourself but there are some weird edge cases that we'll look into but also cube city our copy really cool vulnerability I have five minutes cou that is I've probably hit halfway so I apologize for that but we'll get into the boundary hijacking quickly so executors containers tar binary and look at this link it's really really really cool basically owning kubernetes control planes so at some point a container might be executed so what do we do we just hijack all the binaries so yeah you break stuff in the container what I am gonna jump to is these all in the slides that's how you do it in this case we put

in containers but so what can't what you can do with binary hijacking it might be immediate it might not be immediate it might happen in one minute might happen a month later I've seen a month later where containers of mine in certain systems have been archived and what's happened is then the orchestrator has executed Co commands in my container and provided a token and of course I haven't gotten that token of course you can break systems a command is expecting output a you provide B the system breaks itself and then general blockage and I've just gone the red flag thank you for your time the slides are available on the Reaper and I'm gonna stop talking

before they throw me out so thank you very much