Containing the Threat: Analyzing Cryptomining Campaigns

It's Bogdan Sander and Mihi Vasellesco. I'll pass you over to them now. >> Thank you. Um hello everyone. Thank you for having us here. It's good to be here. Uh today we're going to talk about crypto mining attacks and with a focus on uh containers. Um we're going to keep the rhythm up from the last talk about ETSs and containers. Um my name is Mihi. Together with me is Bogdan. We're both thread detection engineers at the crowd strike. Uh we mainly focus on Linux base and cloud runtime. So we love looking at this uh these kind of attacks. Um the agenda for today is going to be a very short short introduction on containers and container security followed up by

some information about uh team TNT which is a bigger name in the containers let's say uh security space one of the bigger threat actors in the past couple of years and we'll finally of end up with a short demo and we'll be available for Q&A. Hi everyone. I will start with a brief description of containers versus VMs. So the both containers and VMs are methods of um virtualizing uh and if you can see in the first two layers they need both an infrastructure and a host operating system in order to run. The difference comes from the third layer. can see in the picture in the case of VM they need a hypervisor like VMware parallels and

so on and they'll create an entire new operating system including the kernel libraries and application and so on whereas in the case of containers the container engine which can be docker podman or other container engines relies on the host operating system kernel to create containers that share that uh kernel and that kernel provide isolation for the containers. Now let's see how that isolation is performed. There are multiple key features that perform this container isolation. But two of the most important ones are name spaces and croups. And as you can see in this picture, the name spaces limits what you can see and the croups limits how much you can use. And let's see what that

means. So there are multiple namespaces Linux namespaces but we'll just talk about the pit namespace because the principle is the same on all of them. So if you can look here in the right picture at the top we've created an plain Ubuntu container and if you try to list the processes using using ps we can see that we only have the pid for the entry point and also the pid for the ps command. Whereas if you create the same container again but this time specifying the pit equal to host we can see that we have multiple pits including the init the kernel workers and so on. Basically we're are able to see the the pits from

the host and in order to understand why this happens we can look in the right picture. So in the right picture we are listing the name spaces for first the Ubuntu user then for the first container from here the plain one and the in the third yeah in the third part we are listing the uh name spaces for the container with the pid equals to host and can take a look in the first and the third one we can see that the pid iode number is the same for both the Ubuntu and the docker container with p equal host. So sharing or or better said when you create a container with a pal host basically it

means that you share the pit name space of the host and that's why you can see here all the pits from the host. Now moving on to croups. So by default a docker container has no memory uh constraints and can use as much memory as it wants and an attacker can easily create a basic script like this one and effectually launch a for bump. Let's see what happens under the hood. So if you try to list the or cat actually the memory.mmax file for both the croup of the container and also the croup of the Ubuntu um user. We can see here that the value is max. So basically this means that both the user and the container can

use as much uh memory that the yeah hardware allows or have. Now before moving on to the attack we'll go very quickly about the docker architecture. So the dockerjecture has three main parts. It has the client which is the command line interface where you can run commands like docker build and so on. We have the docker host which contains a docker demon which is a running background process that is waiting for commands from the docker client. And we have a registry which is a place where you can store your docker images. The most well known is Docker Hub which is a public one but you also can have private registry that you can host. Now usually

when you install Docker on your machine the Docker client and the Docker uh demon are running on that machine and they are communicating over a unit socket. But in case you need to run both the Docker CLI and the Docker demon or to multiple machines, you can do that and they will communicate over a TCP socket. This setting up this connection is a bit tricky as we will see later in the attacks that we will present. So now that we had some information about uh some of the concepts around uh how containers work, let's try to get up some information about why are these uh why are containers uh targeted let's say. So the main motivation around the

um container attacks are tied to cryptojacking and essentially uh turning a profit. Um so that's that accounts for almost 75 around 75% of the attacks that we know this and the other let's say 25% is split around information gather gathering um DD us as a service or bought this as a service but all these essentially still kind of tie up or focus around uh turning for profit they're selling uh information that they or credentials that they get from the from their targets or they provide access to third parties uh for different uh purposes. Uh now as a volume, so this is something that I pulled up just yesterday from from some of the telemetry that we have.

Uh essentially we see roughly around 30,000 uh events that are related to miners just around every week. That's over a thousand uh events uh daily tied to miners uh and you know anything related to like scripts used by miners. Um mining on itself is not illegal. Um the issue comes when you know the it becomes an issue when people use this um deploy miners on let's say somebody else's infrastructure. um most common uh most common currency that is mined are Monero, Bitcoin, Ethereum. But we've also started to see some some new let's say coins that haven't we haven't previously seen as often such as Verus and others. Um in this space, uh Team Tint is one of

the bigger names. You've probably seen if you follow the the new security news around container attacks. Uh team TNT has been around for roughly five five years. Uh and they've been the say bigger name around this area. Um and they typically do around two or three camp bigger campaigns a year. Um so they started initially compromising like exposed radius services uh servers or Jupyter notebooks that have been exposed and then they kind of transitioned towards uh Docker APIs targeting exposed Docker APIs adding credential theft to their um to their tool chain and finally um they've started adding they started targeting Kubernetes as well. Um usually so one of the last bigger campaigns that we had the jatlin gun uh

the attack flow is fairly um and not just the the jingling gun but fair they fairly use the kind of the same approach uh with docker and kubernetes um they usually scan for exposed docker APIs or do exposed services uh in the case of docker then they would either traditionally they would deploy some alpine or Ubuntu vanilla flavor containers where they would follow up with their own um let's say payloads and then establish persistence with some C2 with some implants such as sliver tsunami and then they would establish then they would start running mass can to find other exposed containers on the internet. Uh aside from that, one of the the let's say the latest developments is starting

to use um their own uh Docker hub accounts where they post let's say prep prepared malicious images. Uh this one has been thoroughly documented. The NML99 account has been thoroughly documented in the past. Um, what's interesting about it is even after the TDG campaign in November, they've still been experimenting with various uh containers and various deployments. Um, and it's actually still up even now. I think the the last activity they had was around two two months ago with new images. Um so let's try to take a look on how they've evol how their techniques evolved and how their scripts their deployment scripts uh started out and how they've evolved during this period. >> Okay. So we'll start looking over their

past init script which is the first script that they executed and their attack is pretty straightforward. They used masscan in order to scan the internet. After they found for for these ports, after they found one of multiple of these ports open, they will use zgre in order to check that indeed docker is running on that service as not something else. And after that, they will issue a docker command in which they will mount the host file system inside /mount. It's an actually it's an alpine plane uh image. Then they will uh change root into /mount. basically means that everything that it's executed after that will be executed on the host file system and then they'll just curl for that init

script pipe it into bash to executed in memory and not leave any trace on the disk. Now another thing that they did is that they renamed common binaries like curl wget like curlbin or other names in order to bypass some naming detection. Other than that they started just doing the same. They started executing multiple curl commands and piping them to bash in which they set up the xmarig or monero minor. They set up persistence. They set up the credentials steer and so on. So nothing fancy. Okay. Now moving on to some present in it script. So the first part is almost identical. They still use mascan and zgrab to find um exposed docker APIs. What changed is how they

create the containers. So now they execute two commands. In the first one they execute a command using a custom monero image that they've created and shared on uh docker hub and the second command they don't crruit anymore in slashmount. they instead install the what whatever they need and then they curl for that for their domain for the TDG in it script and piping into bash and after that from the script they croot into the host file system. Now here this is the their image that is still posted on Docker Hub. And if you look here on the image layers, we can see here in the last two layers that they just download the XM rig minor

archive and then setting up the entry point to run the minor. Now another very interesting thing they started to do is they renamed the XM rig minor to SBIN. They use EPX to compress it. And after that they started infected uh infecting all the running containers with docker exec commands in which they would execute that xmarig u binary for persistence they used sliver now in the past they used tsunami and the infection let's say it was pretty straightforward they just downloaded from their domain and all they always uh rename that uh persistent mechanism installed tmp bio set and sliver is just another open-source adversary C2 framework. Now another thing that they did is they um listed the known hosts file and after

that they executed SSH commands in which yeah they executed an encoded B 64 script. We can see at the top and if you try to decode it we can see that still more of the payload and can also see the TDG in it script the initial script. Now for the cubelet uh for the cubernetes part we can see that the principle is the same as in case with docker APIs they still use mascan on that specific cublet default port and after they find cublet exposed they will issue c commands in order to get the information from that cubernetes cluster. >> Now to take a just a pause before we move on to the demo. Um if you've been

you know if you look at what kind of the tools that they've been using throughout their let's say their activity it's been mostly bash scripts and some let's say off-the-shelf tools for scanning or pivoting and whatnot or open source tools like sliver etc. So there's nothing let's say super sophisticated that needs you know uh reverse engineering or to get a to understand how they they work. They're just um they are scanning finding anything that is uh potentially exploitable or vulnerable and they can that they can easily uh deploy their their their u scripts infect the servers and move on. So that kind of gives us also some u some detection opportunities. Now for the demo we chose to try to implement

something very similar to give a um to get a sense on how easy this is to replicate. Uh so what we chose we start we set up our own let's say attacker control server to uh scan for some docker demons. We set up two two machines. Uh one we're going to infect both of them separately. one with a predefined image that we've created a fake crypto miner. Uh and then separately we're going to deploy again an alpine vanilla alpine uh container and we're going to deploy our own init scripts and then try to grab some credentials and establish some persistence. Um now one of the things um that we kind of recommend we're going to go and and

mention later as well is that we recommend using um security like runtime security monitoring and for both of these demos we've set up uh two both of the VMs we set up with one with a commercial tool and then separated with an open source tool just to get some views on like how what detections we could get and how we can leverage these tools to detect and stop these these sort of attacks. Um, let's see now just a second. And this is the this one. So, uh, once we have identified our targets, we pretty much see the same commands that we've mentioned earlier. timeout secure Docker deploy connect to um to a host that has the Docker API exposed and

just download the the container the image or are not fake crypto image and run that one. And we do this for both of the targets which are on on the lower side of the screen. And we chose to uh present the top so we can see how um the system reacts and what we can see there. So on the left hand side we have the commercial tool. The left the right hand side we have Falco which is an open source tool. Um and now we can actually see the the the entry point for our container running. We set it up to like to run at 50% CPU just to showcase that it's still running. Um on now

let's try to get a look on what detections we can get from this. Okay. Uh so on the top side here we can see that Falco actually gets the detection. It's not going to actually stop. It doesn't have a prevention uh setup by default. There are some options to enable that out of the but it's not out of the box but you will get some information on there's that there's a potential crypto miner running um and you know there's options to see you with commercial tools you know that the the the the minor has started and it was actually shut down instantly. Um okay for the uh sorry second demo there we go. Okay so in the second demo we'll try to

replicate the part with the unit script and the credentials grabber. So what we did is if actually first let me say that on the top is the let's say attacker control terminals and on the bottom we have the targets and in the first here on the top on the left uh we are running a docker uh command in which you into the mount and then we execute for a curl command that downloads that init file and then we execute it and what that init file is doing is adding SSH key to authorize keys. It adds a chrome job and then it's uh trying to grab some credential that it can find on the target. And here at the bottom

we can see here first that a chrome job was added for root in which every minute executed a curl command for chrome.sh file that executed into bash and here at the bottom we can see that a public key was added in the authorized key. Also on the top right we can see that there are some four files created that contains credential for the docking login and also for the etc password file.

Okay, here at the bottom we can see the detection in Felco and we can see here that there was some uh keys that were added and we can see here that we get a detection almost every minute and in the commercial uh tool we can see that this process was killed that was trying to add a crown. Now let's see why the these kind of attacks are easy to do. Well, if you try to Google for top 10 Docker security best practices, you possible it's possible that you'll yeah find an image like this one. Well, the problem is is yes, these are good practices to do especially for developers that manage the containers, but the sad part is that the attackers

don't care about them. And why they don't care about them? Well, because when you expose your Docker uh API, basically you are giving the attacker entire control over your Docker infrastructure. So, for example, let's take one I don't know, let's see, um the lint docker file. So, it says that make sure you scan your Docker files, give it a try to lint. Well, this is useful for example when you're a developer and you try to create um secure docker container based on a docker file. But in case of attackers, they have total control of how you they can create your uh their containers and they practically bypass every every one of these good practices. But there is one of them there that is

states that don't expose the docker demon socket. So basically uh if you remember the architecture that Mihi presented their first step in the kill chain was that they were scanning the uh internet and find to find in order to find docker demons or expose docker demons. If you don't expose a docker demon basically you just cut their entire yeah kill chain and they can't infect you anymore. Now it's possible that in some scenarios you may need to expose your Docker API and in case you need to do that uh what you can do is docker provides a method um in order to authenticate the requests based on TLS certificates. So in case you really need to expose your

Docker API, you can view certificates so that in this case the attacker can no longer um run their own containers. Now another thing is to use runtime security monitoring. You can use open source commercial whatever you want. And another thing to mention here that is often mislooked is running docker in rootless mode. And we'll go back a little bit to name spaces and here we have two example on the left. So we have we create a container here at the top with docker and here at the bottom we create a container with podman which is another container engine. So here in the first picture first we list the user uh namespace of the Ubuntu user then create

the container list the the container um user name space and then we try to root into mount and do a ls root and we can see that we have we can see the snap for that. Okay. Now, if you take a look of the the over the iode numbers of the user name space, you can see that they are pretty much the same. So, basically, when you threw a crruit/mount, you just basically perform a container escape. Now if we go here and look over the podman example and do exactly the same thing, we can see that when you try when we enter or land inside the container and try to list the use the yeah the

container user name space you can see that um the iode number is different the than the Ubuntu one. And if you try to escape the container with C root, we can see that we get a permission denied. And basically why this happened and why is this difference is because of Linux. So if you look right side picture and if you try to list the user namespace of the Ubuntu user and after that we switch to the root user. If you list for both uh users the namespace we can see that it's the same because in Linux there was by default only one user name spaces. So that's why in the case of containers or sorry or docker container it works

because by default when you create a container it's just using the Ubuntu user namespace. In case of podman it creates another user namespace and that's why you can't escape it anymore using this technique. So finally as I think the takeaway from from this presentation is not necessarily that Docker is bad or there's u uh that you should use some security tool we any we one of the things that we wanted to for you to take away is that security runtime security tools are important and uh whether you use a commercial or or an open source one that's absolutely fine as long as you get some information from uh from what's going on in case of attacks. Um,

and the other thing I think is that there's a big difference in between like best security p good security practices from a developer standpoint. Uh, limiting resources to containers or making sure that it's not the that applications don't run as root. But once you're able to escape from that container by default, you're going to end up in a in a privilege escalation situation. And that kind of uh the approach for best the approach for best security practices for developer are different for let's say uh defense and that's something that should be taken into consideration. Uh and finally, one of the aspects that we found fun was that during the demo that it took us roughly a couple of

hours uh to set up the whole infrastructure for the demo, set up a VM, uh build a fake crypto miner, credential theft and all that and target some of our let's say um some VMs that we already knew were going to use for the target. But that's a couple of hours to set up an infrastructure for attack uh once let's say a new vulnerability arises. Yeah. So let's say you don't have a Docker XPI API exposed, but you'll have something like Reddis or there's a CV, a new CV or whatever in some application that runs in a container uh widely. That's a couple of hours until someone can set up their whole shop to target

you. If you don't have something to protect yourself, again, runtime security tools, open source, commercial, whatever, that gives that leaves you in a pretty bad position, I'd say. So yeah, thank you for uh for attending. If there's any questions

>> uh does anybody have a a question at this time for the guys? No. Okay. Thank you. Uh they'll be around. Um and you can also reach out through the discord uh channel as well. So guys, thank you very much. [Applause]