Container Crisis 3 More Containers More Problems

option there I've been doing security for about six five years now and done a couple of podcasts as well um that have been either a guest or I've been writing for them and doing research for them so I've been across the board just all over the place I guess um so this is the agenda for the talk we're gonna do a rtfm on containers just as a precursor I just want to gauge the room here before I really go into rant mode some possible uses that I've used it for in penetration testing and also intelligence work or open source intelligence I'm going to show you guys how to actually hack in and out of containers a little bit and then the big thing of the intelligence portion has brought is actual botnets and also give a preamble on how bows work with the addition of where we're at today because I originally did this talk in 2019 with the first blog post I did about it in 2018 so it's been four years now so um why do we need to care these are the initial numbers from the original talk that I did showing that 25 of Doc of companies have adopted Docker in 2018 and you can I find it funny that a bunch of them have abandoned them in 2018 but uh they all came back uh this number has only gone up and what's also funny is as that number has gone up as of like last week datadog put out another container overall here's what the data is on it how many companies are using it thirty percent of hosts are running unsure Point unsupported container D versions which is the basically how containers get created in the operating system this goes across not only Docker but kubernetes as well and I've got a new number that was really scary when I read that last night um and there's also probably some pretty sensitive stuff on there you've got you know your normal web apps you have red s databases elasticsearch sorry I'm calling you guys out even though you're a sponsor apparently um it's not it's not you that did it it's your users no okay um so let's get into the rtfm um who here actually has used um Docker or knows what it is okay who's got it in production okay okay so what is a container it's like a virtual machine but it isn't this is the uh doc or the graphic that Docker put out to show us like yeah it's just the application running inside of its own thing you don't have all that bulk here's how I like to think about it it's kind of like a container is a Russian nesting doll um where you have uh everything has its own defined space it doesn't need all of the extra infrastructure like you would with virtual machines where virtual machines usually have this big Enterprise server and you have everything all the file systems divisioned out and allocated out for the rest of the uh thing and all that um like I said well I asked that question where it's used is in production just looking at children you can find a lot of places are using Docker and you can find out what kind of infrastructure they have just from an open source perspective so developers use it a lot too just to kind of It kind of fixes the and why I first heard about it when I was still a web developer was we were using it at to fix the it worked on my machine problem we were also using vagrant beforehand but then other computers were not uh RAM intents as what we were using so what it's great for for what I've used it for um it's great for running tools um I hate dependency hell um so I love that it handles all the dependencies I bake that in and it just works most of the time um segmented networking for reverse engineering a lot of the guys that I talked to when Docker first really blew up back in like 2017 that we're doing reverse engineering for ransomware and stuff like that they were actually exploding um the malware inside of containers because they all had virtual box or virtual or VMware Flags to check if hey am I running in a VM and then it would be like no you're running not in a VM we just don't know what this is in the malware at least and also development of tools for across OS testing um which is great for me when I'm doing different platform shelves and all that fun stuff um so just a very simple example this was my first project and I didn't throw this up here as a kind of tutorial this is your Docker file it's fairly simple so what I did is I wanted to replace my Cali VM so I made a Docker image five lines of code or sorry yeah yeah five lines of code and I had top 10 Cali tools I had the Cali uh distro running and I also had metasplate up and ready to go that was 60 gigs in a virtual machine and it comes down to uh four or sorry five here so that to me when I was living uh in college and I also had to redo a bunch of stuff on my machine for classes and I got it I was running out of disk space it was great when I wanted to actually hack stuff in my free time so what I've done in professional work is I've just kind of just picked a tool or an attack so if I have something like a cert.sh where we wanted to do a bunch of different certificate Intel stuff if we wanted to do fishing Go Fish did a bunch of their stuff into container infrastructure as well which we then were able to pipe into evil engine X um and then also running c2s on infrastructure that we would have in the cloud so I could automate that and not have to worry about all the weird configuration and dependencies I just have containers for that so let's talk about actually hacking into containers um first thing we need to do we need to get into onto the container then we're going to check Privileges and then we need to figure out a way to actually persist you know the typical hacker methodology Discovery exploitation persistence Expo and all that so like I just said there um there's two different ways to kind of do this most of the time if you're going to be looking at getting onto a container or if you get onto a container you're probably going to get in through a web app 90 of what's on production environments is going to be web apps or something like that that has an external presence so if you're doing an external web app assessment you're going to probably run into something like that that's the most common one and then another thing on getting onto a container is the docker socket and I'll come back to that one because there's a whole slide for that if you want to look more in depth on how to use these for privileged escalation hack the Box a couple of the older machines actually have a couple of you you break in Via a web app and then you have to privilege desk either through reverse proxying out another container's port or actually um interacting with the docker Daemon inside of the machine or inside the container so let's say we get onto a container what are what's the first thing that we need to do the first thing is that we need to check our privileges these are the default privileges that you have in there which means every single container no matter what configuration that a developer has given it has these set conditions um so we can change the root directory from any container at the second the last one there but what's really fun is if a developer does this which I say never do never do this privilege and tact cap at all do the exact same thing within the docker CLI and what that basically does is anyone see the sys admin one there perform a range of system administration and operations kind of broad right so that means you can mount the host file system to the container so this is inside of a Docker machine which is now deprecated um it was a way to run Windows or one run containers on Windows it was just basically inside of a VM and it's just a Docker container there but if you see on that last line it says containers so if you were to find something like this you can just then say um yeah well that's cool that was actually the culmination and I wish I put this meme in my bachelor's project but that was actually the the big exploit in my bachelor's project was I was like oh yeah I just did this that was easier than I thought which then got me thinking again um and talking about that bachelor's project this is where we I pivoted more into looking at okay how can this actually be used because if it's this easy for me to not only get onto a container I mean well getting onto a container is one thing but also just if it's this insecure to if a developer is able to just give all permissions to this container that doesn't bode well for the overall security of any sort of application or network that it is attached to so what I said here at the very conclusion or whatever is I was like botnet that probably is the best way to do it and I hate when I'm right I was not completely ahead of the game here the first iteration of crypto minor mining Docker crypto jacking botnet was um 2017 there's actually a GitHub issue where you can actually I think it's still up where you can actually find the first instance of this but this was the first article that I found after I found a couple of them in my free time um and we'll actually go through how I found them and also do some live threat hunting because it's still there um also at the same time when uh right after I found all this out this was a very bad cve basically it was any container became a privileged container because you could just make it one and then Escape it by interacting with the docker socket like I said Docker socket this is the and what am I doing on time I actually watch on for whatever reason so what the docker socket basically is is you have let's say I have a server up in the cloud I want to be able to run stuff on there and I'm too lazy to set up SSH keys what this allows you to do is it allows you to run a Network socket on your Amazon your digital Ocean or whatever usually it's on Port 2375 port 2376. and what that allows you to do is just tell your CLI Docker run Tech host and give it the host IP and the socket Port that you set up and then it runs stuff there yeah so you remember this image that is an example of all of those being able to run so you can literally just go in and say hey thirteen five six one five eight one seven three run my code on anywhere from anywhere and also unencrypted so one night I'm like this is bad so I start looking around I'm like there's no way that I'm the only one seeing this and one okay cool I'm looking at a Chinese address that's fun um but I came across this one image that kept popping up here and it's this Zulu 2 slash Auto I don't know what you guys I don't know if you guys are seeing that okay um so I was like that doesn't look great because at my point in time I was like you really shouldn't be doubling up that looks odd also it's not a known service it seems sketchy looks sketchy probably sketchy let's go look so Google the username and someone else actually got hacked by it and put an official issue in on Docker hubs or on Dockers GitHub which I thought was funny um so he actually had a Honeypot already set up because he was like I'll just see what what happens uh talked a bit in the issue and then I was like okay he didn't know if what was actually running on it and I was like okay cool time to hack a hacker kinda just downloading it so pull down the image that was hosted on Docker Hub and this is what I figured out that you can do so if you actually look at the documentation you can change the entry point on any sort of on any Docker image so it may have it say in this case it was the entry point was um bin slash bash slash entry but I was like oh okay cool so now I know what code to look at and then I just said okay cool just pop me into a shell don't run what the actual code is and then it's just all the source codes right there which is really funny because then all of his Source code's right there and he's put it everywhere so if we look at in cat entry we determined and it's a very small text and I apologize for that on the very almost to the bottom we can see that it's actually going in and Mining cryptocurrency specifically Monero on the actual box so any of the boxes that we saw oop wow it just went all the way to the bottom hang on okay again and again and again I apologize for that I hit zero and it went yeah QA time um where was I yeah so we knew it was mining Monero we wanted to figure out okay how is it actually working or I want to figure out how it's actually working so there was this file called um botnet which I was like great yeah not not hiding that's cool tour sucks nice also not hiding was his username and passwords for all of his shown I mean yeah that was funny so then I contacted shoden because these are actual these were shown in accounts that he was using to do the exact same queries that I'll show you in a bit um to go through and um just grab an entire list of them go into them and then say okay cool now now they're mine I'm mining Monero so three in the morning successfully takes down a botany yeah that was a fun email to receive that I have archived I probably should frame it at this point okay so he showed up again so this time under the user name Pavlov 32 Auto Okay cool so same name image same whatever same key same basically it was the exact same cone got into got over to shoden he was still using he just basically made a bunch of new Showdown accounts he then throw it up threw it back over to them they took it down it was like great and again this time just recreating the same username of zulu 2 so I was like okay this is not going good told Docker told showed in and I was like okay that went down and that time he actually used the Pavlov 32 username so then I was like yeah you're the same guy even though the code looks exactly the same but now I can just say that it's you so great kept doing it kept doing it and again this time he just redid Pavlov 32 which is great the same code again um actually at one point um I kept tracking him down and kept tweeting out his uh his keys and everything and his passwords that he basically Ford everything so it was 32 000 lines of a base64 string yeah like I'm not just going to pipe that into a shell dude into base64d code and then whatever anyway not very sophisticated I effective not sophisticated so to kind of help out and actually what happened during the whole me talking back and forth with shoden was they disabled the actual um query via the website for about three months so I couldn't see what was out there I couldn't keep investigating or anything so what I did is I was like oh okay cool there's an API let's just build an API thing so then I look at the API and I'm like oh yeah oh I can see way better now um so I wrote a tool called Dalek which was a joke among my friends because they were like yes you must exterminate the botnets and I'm like like all right cool I'm not good with naming things um which um we're actually I'm going to show that um doing it live because and this will get into the the point that I want to make on why it's still an issue so let me see if I can go in here so I have a little shell script here that's just a bunch of the queries and this is all on GitHub you can play around with it and so I'll just cap this out real quick here um are you guys able to see that okay or do I need to make it bigger in the back all good sweet um so I've written the tool so that just takes normal Showdown queries I mean it passes it over to it and um these are the most common ports that you're going to find and open uh Docker Damon on uh so your 2375 which is the default that is set by Docker when you open up the docker Daemon socket 2276 you have to set it and then it gives an SSL um tunnel to work through there 4243 is just like the alternative for some reason that every developer just decided was okay I don't understand where they came up with the number at all it might it doesn't make sense to me but and then um actually kubernetes has um a similar issue especially ones that use the docker container D socket they actually have the same problem and also this is their open dashboard port and you can actually if they have it configured incorrectly or just by default it is completely open so if you see the node is attached there which I actually added last night and I'll show you guys why I added it last night in a second here so if we just run that it looks really like stupid and hackery because I was bored and made a banner that's obnoxious and it's not going to work so we'll see if the internet actually works so it'll go down it'll go up to show it and it will grab everything out it'll actually give you a Json file and then I have additional parsers that actually show me the actual um uh context of those files a little bit more parsed out here so I'll make it bigger again here by the way this is what my life has been at night for like the last four years late at night my wife can attest to that right she just went mm-hmm so this is what the tool will spit out this is the um docker.txt she's just talking up a storm now I'm screwed um anyway uh this is the docker.txt file that gets generated um I put it in txt because I hate reading Json for whatever reason when it's huge when it's a small amount of stuff I can work with that but I was just like Okay cool so I would pull this last night and for Friday November 4th at 1400 UTC or sorry 1600 UTC this is an actual um right here no that's not going to help this is this is an actual actor that I'm pretty sure has and has been correlated by other um security companies and threat Intel companies that is associated with um it's called kinsing I think is the D dot sh or wait no is this team teams TNT no this is can can sing um this one's associated with um Asian State actor I'm not sure which one exactly but it's either China or I think it was China that they put this one on and it's been running for four years and all they've really done is changed up the C2 every once in a while so wherever they're pulling down the script from that that's the only thing that's changed and I've also tracked all that and if you want to actually look at that code um what did I have for this yeah so that tool actually pulled down all of this last night um and then at one point here's the other issue that I have or that I ran into last night my query brought that up if you can't tell what that number is in the back that says that kubernetes had 387 000 different results roughly with that Port open that does not mean that they were vulnerable to what is similar to the docker socket where you can just build a node or put a container image up there that was closer to 711. it's still bad but imagine if um we have an event where um someone figures out hey if this port is open we can just hit this endpoint really really hard and now we create containers on or nodes inside of the kubernetes port that's a big deal um and it's also why I've kind of been tracking it for a while um and I've put up everything that I have ever found in um a GitHub repository that's aptly named docker botnets uh under my GitHub and it's fully public for everyone to go and take a look at um and I've found I think 50 unique botnets over the last four years from different actors a lot of them are repeat offenders there are two right now that basically have set the Precedence for what it looks like to actually have a good kill script because what they'll also do they'll get onto the Box they won't just install a crypto Miner anymore they'll get onto a box they'll kill all of their competition which also gives you a nice detection script for all the other ones and then they'll install a crypto Miner and then go find other ones and then repeat the process so everyone had their own kill script which was really funny so I was just thought one day it's like if I just put everything together allegedly all the kill scripts and just sent it to everything this wouldn't be an issue but that's illegal because then I would be hacking so on something I do not own which is bad so I'd like to put um for this is kind of like