BsidesLV 2025 - Ground Floor - Wednesday

[Music] Heat. Heat. [Music] Heat. Heat.

[Music]

Heat. Heat. [Music] Heat. Heat. [Music] Heat. [Music] Heat. [Music]

Wow. [Music] Yeah.

Heat. Heat. [Music]

[Music] Heat. [Music] Heat. [Music] Heat. Heat. [Music] Heat.

Heat.

Heat.

[Music] Heat. Heat. Heat. [Music]

Heat. Heat. [Music] Heat. Heat.

[Music] Heat. Heat. [Music] Yeah, [Music]

[Music] down. [Music] Heat up [Music] here. [Music] Yeah, [Music] down. [Music]

Yeah,

[Music] Heat. Heat. [Music] Heat. Heat. N. [Music]

[Music] [Music] Heat. Heat. N. [Music] Heat. Heat. [Music]

[Music] Heat.

Heat. [Music] Heat.

Hey. Hey. Hey.

[Music] Heat. Heat. [Music] Heat. Heat.

[Music]

Heat. Heat. N. [Music] Heat. Heat.

Heat. Heat. Heat. [Music]

Heat. Heat. Heat. [Music] Heat. Heat. [Music] Heat. Heat. N. [Music]

[Music] Wow. [Music] Woo.

[Music] Heat. Heat. [Music] Wow. [Music] Heat. [Music]

Heat. Heat.

Heat. Heat.

[Music] Heat.

[Music] Hey, heat. Hey, heat.

Heat. Heat.

[Music] Heat. Hey. Hey. Hey. Heat. Heat. [Music]

Heat. Heat. [Music] Heat. Heat.

[Music] Heat. Heat. [Music] Yeah, [Music]

[Music] down. [Music] [ __ ] Yeah. [Music] Yeah, [Music] down. [Music]

down.

[Music] Heat. Heat.

[Music] Morning. [Music] Everyone can hear me. >> Uh welcome and thank you for coming to Bside Las Vegas uh ground floor. This talk um is called hardening containers with SECOM and it's be being given by Ben Hsburg. I have a few announcements before we begin. We'd like to thank our sponsors especially our diamond sponsors Adobe Aikido and our gold sponsors profit and run zone. It's their support along with other sponsors, donors and volunteers that make this event possible. Uh these talks are being streamed live except um and as a courtesy to our speakers and audience. We ask that you check to make sure your cell phones are set to silent. Um if you have a question, please use the audience

mic so that YouTube can hear you. Make sure to point at the mic. Make Oh, sorry. I'm reading what it told me to do. You may be asked to make announcements. Oops. Sorry. Uh, as a reminder, besides Las Vegas photo policy prohibits taking pictures without the explicit permission of everyone in the frame. These talks are all being recorded and will be available on YouTube in the future. Um, thank you so much and without further ado, >> good morning. Sorry, good morning everyone. Um, thank you for coming. um to this talk. Um I'm going to be I'm will try to be easy on you this morning. You know, it's early morning for first talk. Uh I see that most of you still

have your coffees on which is great. Uh uh hope that uh um this will work smoothly. I'm going to give you uh two live demos. Uh you know, let's hope that the demos will work. If not, then we'll switch back to uh recordings. So, uh, those of you who don't know me, my name is Ben. Uh, I'm a co-founder and CTO at a cloud, uh, security company called Armo. Uh, used to be, um, worked at different places before. I'm working in the security industry for, you know, more time that I want to admit. uh but coming from uh from a white hat uh uh background and going into uh more to product and development uh um part of the industry. Uh we are

going to talk today about um an open source project I'm maintainer at. It's called Cubescape. It's part of the cloudnative computing foundation which I'm an active member of. Um and yeah let's go in. So uh among you first of all who heard before this talk about sec comp who's using actively sack comp in their products. Okay great. So actually this is really the talk is about. So the talk is about what is SEC comp and why it is not really widely adopted and used and how this can be how we can turn SEC comp into something that's more usable in the at least in the container and cloud uh security sphere. So I'm going to talk

today about um the you know what is SEC comp going giving an intro for those who are not you know who doesn't know what it is uh and then we'll go over how it can be used uh for prevention uh in uh in um in containerized environments and then I'm going to show you a way to be a to make it much more usable. uh uh using application uh uh observability and understanding how applications are behaving and then applying the right profiles for the given application and I'm going to do uh a demo uh I've come with a real uh exploit in a real application to showcase you how it works. So um so sec comp is actually um rather I

would say old feature in Linux kernel. It's uh I think I haven't checked it precisely but it's been around something for nearly 20 years in the Linux kernel if I'm not wrong but even if it's less than not far less. Um it is it was born in order to be able to build sandboxes uh process sandboxes uh uh uh that are running on Linux kernel and uh uh it is uh uh it was built for two purposes uh originally. One was to build sandboxes around u things like uh browsers that are you know running less trustable code and was were prone to different uh uh exploitations before and therefore there was a need to create a sandbox around uh

around these applications to limit them. So even if there is an a real uh real exploit on them then the attacker won't be able to proceed uh uh to the next step. Um and uh you know the second reason later become is to limit uh also system calls that may be problematic for the kernel uh in the sense that if there is a system call that there that has um a known exploit in a given version still the uh the defense can uh use second to limit cost to system calls if the application doesn't need them. So uh the original history was that again uh 20 years ago uh to to create uh uh um

processes that are limited only to four system calls. Um the way work is that a process that when it started up could call into different system calls and just limit itself from any further uh uh system calls like going down in the in in the privilege scale. Um these four system calls were read, write and uh exit and close. Uh so in general the idea was that I can create a sandbox process which is just like you know uh talking doing some basic IO uh maybe closing some uh uh file descriptors and then it exits. Again think about the you know the the the the use case of of browsers uh and executing JavaScript in the

browsers. Then after it come a a more complex uh uh um version that was using uh BPF which later become you know eBPF to like define much more fine grade rules around what is allowed and what is not allowed for the process to to do. uh and creating you know very complex scenarios even even like just not not simple firewalling scenarios but checking into the arguments and so on. So uh SECOM became much more powerful uh in the early uh 2010s and then uh when the containerized environments started to come up, Docker came up uh the guys at Docker came up with a great idea to create a basic profile which we are going to talk about today. um that

limits um the system calls of a containerized uh process uh that it can do um and uh generally protect uh you know not just the application but in general the the the host machine that runs the container uh from uh attacks and uh and you know as containerization evolved Kubernetes itself uh uh which is the de facto orchestration tool today for containerized environment uh also started to support five years ago uh uh uh fine grained uh uh uh second profiles. So what's the present state? So the present state in general is that uh there are like pinpoint usages of of SECMP again uh browsers uh very specific uh uh applications where SECOM is used. Um I've seen very uh um

very interesting usages as well for secion to uh also to intercept system calls and I've seen uh great interesting things but in general uh there the security industry or parts of the security industry know sec for uh for Linux hardening um and it's great but in general in the main you You know in the most places where our Linux today is used and used Linux is used for very broadly obviously in in the cloud um it is much less used. It is like very very rarely I see companies are are taking advantage of of SECOM to to protect uh their environment. And this is the thing I I I you know we are trying to uh uh to change and to uh

to make SEC comp more usable. So why it is interesting why SEC comp is interesting in cloud environments. So SECMP is interesting in cloud environments because um because the main threat model of uh of cloud environments especially if we are talking about containerized environments is twofold. If we are assuming that there can be some malicious code that is running inside the container, the attacker can do two things or advance to two directions. Uh one is to try to exploit create an exploit on the kernel and somehow you know uh uh uh leave uh the confinement of the containers and take over uh the the VM or or or the Linux host that is running the container. This is one thing

and usually when this is done, this is done through some kind of um um system call that the attacker is able to invoke with specific parameters and then it enables it to do a container escape uh because exploiting some kind of a vulnerability that there that there is in the in a kernel that wasn't patched or maybe it's a zero day and no one knows that there is an issue there. That's one thing. The second thing is that the attacker might try to do some lateral movements maybe not trying to attack uh uh the kernel itself but still uh um the workloads are are rather you know li unlimited in what they can do in their environment. So

they can uh talk over the network. They can invoke uh uh uh different network calls in their environment if there is like no network policies around that are super limiting uh files that are written uh uh uh on the disk and so on. Maybe environment vul. So there are like plenty of things to do later movements too. And these two in this threat model, you know, attacker can either come from, you know, exploiting uh uh uh uh exploiting a vulnerability inside the container and coming from the outside world and it comes from the public internet or it's just like adding itself into the supply chain and you know the container is environment starting a container that already includes uh the

malicious code and uh this is our basic threat model. So the way uh I see it is that one of the ways that SEC comp is is would be really great for um for containers is to limit the availability of of an attacker to what it can do to to you know the progress attack. Maybe not limiting the actual exploit of the initial uh attack vector but to like create these lateral movements or or trying to go over to the kernel to continue an actual attack. And I think that this is where SEC comp can be super super useful. So let's talk a little bit about how uh uh sec works. So sec uh sec for uh for containerized

environment environments is uh um is very working in a very schematic way. Um as I told you before in general secump if you are programming through the Linux API you can like create very very very specific uh use cases and implementations of limitations but uh if you are you know higher level abstraction layer like a container runtime you don't want to allow all the users to all these features because it's just like super complex to do and it would require uh uh you know deep deep knowledge of of of programming. So the way the that Docker originally defined second profiles and this is like a very big difference to know that in between the sec API of Linux kernel and

the second profile of the container runtime. So the second profile of container runtime is defining this very schematic way of like there is a system call there is it's very it has very specific arguments and like just tell me what I what should I do if this happens. So if there is a I don't know um a fork system call um then you know block it. uh don't let the application call into fork or clone and you know prevent it to to to create new processes or uh if there is um close uh uh system call on file descriptor 5 um then just like block this system call and this is how uh uh the schematic uh uh sec profiles

for containers are looking like they have this like list of of of system calls but optionally arguments and and what to do with them and this is how they Look, so what are the actions? Um, SECOM today uh enable you to do different responses in case of different uh uh uh system calls and arguments. So um the most basic is just to say allow let it go through don't care. Uh if you are going from lowest priority to upper which is like from the the bottom to the top um you can ask secom to to log this event and then it will be written into the kernel logs. Uh I will tell you in a

minute why it is super uh unusful in in cloud environment but we'll get come back to that. uh there is this notify option when uh you can hook into uh uh create a hook into Linux kernel and get notifications about a system call event. You can uh configure the Linux kernel that if there is such a system call then just return an error code like not allowed uh a and you know the application continues to run and it will just get an error on this system call and that's it. Um there is this trap signal. You can also define secum to to throw a signal on the process. Um it is again it is for applications that

are are are very uh aware of how the sec pay API uh works in the Linux kernel. Um I've used it few times but it's just like for very niche purposes. And then obviously then you can have the kernel to kill either the the caller thread or the the whole thread group the process uh uh uh in this case. So kill the application uh in general if there is this the invocation of the specific system call. So you have all these options. Now I wanted to return to uh uh the logging option. So it you're thinking it's like super nice to like define that if there is an application that is like calling to a very specific system call and I

just have a log event about that. Now the problem is that most cloud environments it is like really hard to create this feedback loop because these logs are not really collected anywhere. There is no standard way to collect it. There's like a very specific implementation for each cloud environment cloud provider. they are not at the upper layers of container runtime Kubernetes it's just like these logs are you have no unified way to access it and it's like a shame but this is what it is um so again returning to the point that I had before of the difference between the uh system call uh uh uh uh uh API and uh uh of of sec and the sec profile

itself. So the way you're working with SEC comp in case you're using directly the Linux kernel API is you know you have like two u mostly that you have the sackcom system call itself uh which is like a has a man page of which is like super long uh you have including BPF instructions and everything it's it's hard complex uh uh most people you know need to invest a lot of time to learn it on the other hand in the containerized environment you know to implement. It's not really a programmatic implementation, but you're just defining this JSON uh configurations of like what I told you before that when this system call happens with these arguments, what

to do? That's it. Um the format is uh uh uh sorry uh um the complexity of of these two things are really really different. But I'm going to return to this because I could really really say that it's really hard to to to learn the sec API. You it's like any other kernel feature. You have to really learn how it works to be able to control it. Uh and the JSON file is much more simpler. But having said that like it's today it's really hard for people to understand even the JSON file because uh because of other reasons. So um so we'll go come back to this uh uh a little bit later. So um so how this JSON file looks uh

sorry for the small font. I hope that that you can still see something here. Um but again it it is like this super thing uh list of uh you can see sys calls uh items for every item there is a string for uh for list of the system calls and then the action to take. So uh you have the uh a specific system call for example close uh you can define the action to take you can define the argument filtering of I want to apply on this uh uh this action only if there is a given argument to it. Um the argument uh filtering is like super complex and uh because think about the following

things you would say what kind of argument filtering I would like to implement like obviously I wouldn't like the anyone to open a file called /cd shadow because I just want to prevent it now how the kernel sees this string how the kernel sees the the string as a pointer as a pointer to username uh uh uh to the user space address and like sec is not good at it. It you cannot read uh uh directly this and even maybe with epf you could uh uh you could read that but it's like super complex not something that this JSON file can can handle. So it's it it's uh it's you know this argument filtering is

is just partially useful. So if it's so cool, why is it not used? So the main thing is that as opposed to you know I guess you came to this talk because either you know Linux you know Linux security you heard about kernel uh uh you're interested in and you heard what is a system call is uh but I it's hard for me to break you the news that most people in the world who are in this industry doesn't know really what a system call is uh uh and uh uh and people just don't understand that developers uh uh not understand this notion like think about it I'm uh and I do really trying to say this without you

know uh without any hint of disrespect but think about the Java developer uh if you ask him okay what kind of system calls you're using in your application like he's going to be like well no idea um so it's it's it's really really hard to like create an environment where people are aware care of what system calls they need in their application. So think about a containerized application. I ask you today that for example you are using a super simple application called elastic search in a containerized environment like what system calls it use then the next question is okay which version of elastic search which version of of java runtime and it's it you sort

of think it's it's becoming super duper complex uh uh discussion and and and then afterwards like okay I'm as a security person. I'm creating this environment when I'm limiting system calls and for some reason I it looks like the container is running. It's not crashing. Is the application working? Do we know that the actual application is not broken? Like because we are limiting something that is there is that is hidden behind us. So it's it's it's really really hard to to understand as a as a security practitioner. Okay, that we we are not breaking functionality. And now I'm I'm returning to this notion of like SEC comp has that that logging uh blocking system calls that are

blocked. And the problem is that since we don't have a reliable or unified way to to understand that we've blocked something uh in a in a containerized environment, it is really hard for us to understand that that that we broke something. So it's it's very complex. So now how it's you know tapping into into the ecosystem of uh of containerized of the containerized world. So as I told you before, Docker when it came out and and all the other um container runtimes that came after uh um just inherited this property that it when you are running a container, it already attaches a a default basic profile of sack comp. So if you're and I'm going to show it to

you in a demo in a minute. Um so when you're running a container with docker just docker run or uh uh podman run or uh or whatever um it is limit there is a second profile active behind the scenes uh will which is like I I do think that you know whatever you think about docker from my perspective as a security person I kudos like I I I really love when when a vendor comes out or or project comes out and they are just like trying to be really strict on security uh and it's even like works uh to some extent. So it's it's great. On the other hand came Kubernetes which is like you know um I

would say my home team u uh I'm contributor to Kubernetes uh uh uh and work and you know as part of CNCF it's it's our project but I have to tell you that the so that the less good part is that when Kubernetes came out you know when you're releasing the first version you just want to make everything work right. So what's the first thing you're disabling? Secmp security. No security. So Kubernetes uh uh uh disabling sec in the container runtime by default for every workload you're running. Uh so you're not inheriting the good things that Docker came with. Um and reintroduced in 1.2 K reintroduced the support. So you can define uh uh the

basic second profile or you can bring your own. So um so you have these three version uh three ways to work in in in Kubernetes. Either you are running in an unconfined environment which is a no sack comp. Uh you have the default profile which is the basic profile of of docker which is by the way a little bit changed over the time but it's not super interesting so I don't want to go in there. Um, and you have the local host profile when you are just like bringing your own sec profile into Kubernetes and you're running a container with a custom second profile and praying that it works. Um, so let's talk a minute about the

basic built-in profile. Uh, so actually it's really cool because um, it blocks only you know around 50 system calls. again and the uh it's I don't have the specific number because it changes over time. Um but it blocks out a lot of system calls that were usually really not used in the containerized application. Uh think about you know uh mount uh pivot truth and stuff like that which is usually in the normal application. It's not like system calls that are are used but they are like really good things for an attacker to use right. They are like system calls that are also they were prone to different attacks and and also it's just like something that the attacker would

love to have especially unshared to to exit the containerized environment. So it's they are working really smoothly. There are a few uh applications that don't that are not good with those but in general uh uh uh but you know most applications like 99% of of the simple applications are working good mostly I could say same about databases there are few exceptions with that because different uh file uh system management system call related system calls that are like just making problems with few database setups but in general it's working and there are like also very very specific examples of of of well-known CVE in in the Linux kernel uh that are just like not exploitable if

you have a sec by an attacker if you have a second prof uh the basic second profile in place now I'm bringing you this really really super nice screenshots about custom profiles these are super cool again no one understands really what's happening here like you have just like this list of of system calls that are are are allowed and and uh and that's it. The reason why I wanted to show you this slide because it's just like I wanted to give you the feeling that it's complex. It's like you know to understand what's in here you have just like a list of system calls and that's it. Um and it's it's hard. So I'm going to show you the first the

simple demo. I just wanted to do a little showcase of uh of the of how the basic profile in Docker works and what kind of things it prevents. So I'm going to run two NGNX uh containers uh one with the basic profile and one is unconfined. Um I'm going to try to create a child name space uh in both of them. Um and I will try to access the kernel key ring in both and I will going to show you that the server is still running. Now sorry for uh let me go over to

okay. Okay. So let's go over it. So let we are starting the two uh uh um containers. So you can see here that the first container do you see this right or should I >> okay I will try to let me try to go over.

>> It's colors. Let me >> maybe Where is the white? >> Is it there?

This is experience guys. Um so we have started one container. Uh uh this container is running without confinement. I needed to since this is not the the default setting. You know that I needed to tell docker to run it unconfined. Uh I'm running the you know the normal container which just doesn't have this option and it's running by default. Um, so I'm just in installing the Qutails which is like less of and hoping that the network is working.

Wow. Okay. This is the nice part of the live demos that even you're [ __ ] up with the with the step that you didn't think that it will go wrong. Um, let me try to see that network that works is going.

Let's give it a second and we'll see if it's able to download.

Okay, let's switch over.

I'm switching over to my phone. Restarting.

Let's restart.

Yeah, we'll come eventually we'll come over all these problems.

Okay. So what you can see here is that I'm checking that whether there are any sec filters applied. None of them in the unconfined container. I'm trying to create a a new Linux name space. I'm was able to create it. Uh I'm trying to access uh um you know the the key ring. I'm able to create it. So on the other part um let's see if second profile is enabled. Uh you can see that there are like second filters installed. Uh I'm trying to uh create a namespace. I'm blocked. Uh I'm trying to access the key ring. Unable to do this. Okay. And I also showing that actually engineext is working. So that's uh that's the third

thing I wanted to show you. So now returning to our discussion. This is guys guys there there's going to be more interesting things guys. So, sorry. I like I even thought of removing this demo like I just wanted to show something uh uh uh really short. But in general um so the problem is that uh uh u um with the basic profile that there are like a still a lot of things that can be exploited with the with the basic profile and there are a lot not a lot but there are applications that are are not working simply with basic profile. So we are like you know it's a like a glove that one globe that try to fits

all like it's not working everywhere. So um so the question is like if I'm trying to go over and build better tailored uh second profiles what can I do? How do I know which system calls are running and and which system calls are not? So there are obviously like you know uh tools to understand and list it pra. These are like very problematic to use in a containerized environment. Um it's also if I'm trying to create for a for a very specific application for example I told you elastic search uh to create a second profile that it's tailored to it. It depends on the actual configuration of the of the application. So again, I'm getting back to the same problem I was

before that uh I'm trying to create a generic two generic application profile that will allow a system call that shouldn't be there. And uh what again the issue that what if I'm blocking something that that I should be blocking according to profile but in general it's used. So I'm I have a problem. So the whole thing was built up to this point to bring you that okay there is a future there is a future here to uh uh uh to build upon and um this is one of the things we've tried to uh to solve in the cubecape open source project. So just a word because uh uh uh uh I we don't have time and I want to go to

demo. So in cubescape, cubescape is a Kubernetes security platform uh supporting for posture things for vulnerability scanning, configuration scanning uh to supporting generating network and sec profiles uh that you will see and runtime uh detection tools if you want to uh uh uh get to our website uh scan the QR code uh and then if not it's cubescape io so you'll find it. So the way Cubescape is working, Cubscape has this node agent that is running on every Linux nodes and uh and u um and it just sees all the containers that are running and sees what system calls they are invoking. So it it really learns how the uh uh the container behaves and since it's logging the

system calls that each container uses it can like create this list o of per container of the system call usage and we are using ebpf to do that. Uh and what the the point is going to be here that that when we are have these uh um Kubernetes API objects that cubescape creates that we call application profiles that are logged per each container the application behavior. So it logs the the processes that are running the network connections the file access and all these like good things that the security engineer would need. Uh it also logs the system calls. So what we are going to do is we are going to use these application profiles objects to turn uh uh the s list of

system calls into seccom profiles automatically. Uh what I'm going to show you is I'm going to showcase you the uh uh the the power of this through a CV which uh uh I I prepared for um uh for this demo. It's it's in the change detection project. uh uh u that CV is enabled a serverside template injection attack. So actually um I can uh through a uh ginger template I can inject Python code into the server side and I can run anything uh from the client to the server and I'm getting arbitrary code execution of the of the container itself. Um so what I'm going to show you in the second demo which again hope it will

work um is I'm going to try to inject code that steals the Kubernetes service account token which is if those who don't know every Kubernetes container by default has this uh service account token mapped into the container which uh enables the container to authenticate itself against the Kubernetes API which is think about it as just an API key that is located in the Kubernetes container. It's just like a cool thing to to still because afterwards you can talk to the Kubernetes API using that. So what I'm going to do I'm going to show you uh uh the exploit itself uh without in an unconfined uh container then we'll generate a profile based on the application behavior apply that

profile restart uh the container and then we'll try to reexloit it and if everything works we are going to block uh the actual exploitation itself. So let's go over and fingers crossed. Okay. So it is uh the things are up and running here in uh in my Kubernetes cluster. So I have uh uh pre-installed uh this change detection uh application which is the vulnerable application and cubecape is running here in the background which I will talk about a little bit later. Now let's do an exploit. Um so the way it works I'm going to talk with this uh service uh uh through uh uh port forward which is just like a technicality to for me to talk to it. So

I'm trying to exploit the actual uh uh SSTI vulnerability. I need like different I've taken the publicly available PC and turn it into this demo. So it's not the credit is not coming to me. Uh I did some just small tweaks. So I'm obtaining uh uh u the token uh uh CSF token. Let's hope that the website web hook site is work working because I wanted to show you what we are stealing. So as you can see uh I've seen two posts maybe it's from yesterday. Let me try to see I'm exploiting now the actual payload and payload is sent successfully. What you can see in the actual payload here is this Python code hidden inside the ginger templates. So

you can see here that I'm trying to open injecting code to open the service account token file and then post it to the web hook site. Uh let's go back and you can see that the web hook uh the the actual service account token was taken from the container through uh uh through this attack. Um okay, that's it. Okay, you've been pawned as as usual. Okay, now let's go over to uh uh the protection which is the protects sh. So what I'm doing is um I'm retrieving this observability object which I told you um which is the application profile that contains the system calls. I'm using cubectl get application profile uh saving it as a JSON file. uh then

running a small script to convert the list of the uh of the profile to uh an actual application profile CRD that cubescape knows to handle. I'm deploying it. Now I need to tell the Kubernetes deployment to use this profile. So I'm going to patch it now. Uh and the actual profile is set up. Let's see that the co pod is coming up and not crashing. Okay, so it's up after the I've added uh this the new second profile. Now let's try to exploit again the CVE and now fingers crossed I'm running the same script. So

doing the all the technical parts of the exploitation itself.

Okay, let's try to see in. Okay, so we don't have any incoming data and let's hope that it's going to work again. the payload itself printing out and now I'm executing the attack. I've sent the payload and nothing came in here. So actually the exploit didn't work because it caused uh the exploit during the exploitation and I've created I've used the system call that I need to use a system call that wasn't available in the profile and the profile blocked it. So the exploitation was stopped uh uh uh during the middle. So and then just to show you like good side effects that um if I'm coming and I would try to open a shell in this container

um for some reason I'm allowed to right now which is surprising but in general most of the cases h most of the cases I would be blocked. That's strange because I chose it yesterday. So eventually there was a fluke in the demo, but in general that's the idea. So let's go over to the next slide. So just like a few alternatives to to to sec profiles. I'm not saying that second profiles are a good thing in the world like the best eventual solution that you have. uh um but uh they are there they are in the ecosystem of the containers. So trying to introduce any kind of other uh uh uh solutions and I by the way I'm

one of the I love the landlock LSM. Anyone who wants to read something interesting it's it's it's a really cool project. Um all the other are really hard uh to implement in a containerized environment. Up armor and sc Linux are there but they are like really hard to move like they have the same problem as sackcom that only you can mostly you can only work with with predefined profiles and it's it's super hard and it's either too permissive or or killing your application. So it's not an easy world but in general sec can be a really great way to to to protect uh and harden uh containerized environment. Uh and there are tooling here and anyone's who wants

to contribute to the open source project and play around with it, you're welcome to join our uh uh the cubecape uh uh uh Slack uh come to our website uh and and you can see where you can join the community meetings and the Slack channel and in general it's we are really trying to like gap the usability problem around SECOM profiles um uh and uh and make it usable because it's it's really important to harden these environments. That's it. Um, thank you very much. And if there are any questions, I will be happy to take [Applause]

um I don't know how is the questioning working through the phone.

Hello. Thank you for your presentation. So, I have a question. Um, how can I justify to my CISO that I'm going to have to spend several hours with developers securing my containers when there's also a possibility that some of the CVS as you mentioned they are not even covered by SECMP? Is it worth it? How can I justify that? and the level of complexity that it comes with it with that. >> So it's a great question uh and thank thank you for for asking that. So uh the way I can explain to CISOs and I'm I'm on the vendor side here uh most most of the time when I'm talking to CISOs. So one of the major problems that CISOs have

today is that the overwhelming number of CVS they need to somehow manage. And uh and by saying that SEC comp is is like a way to uh uh to uh to prolong the time that you need to invest into updating the actual applications. when you there is a CV out then you know you have 30 days of of like patching that CV. But if you have a second profile and by the way uh I I think that most compliance frameworks are also uh uh uh supporting this claim that that you can prolong this 30 days to even 90 days uh if you have these additional protections in place. So the way to to uh I I think

to convince a CISO in this case is really to say that look one of the biggest things that hurting you is the overwhelming number of of vulnerabilities that you need to patch and you need to have the so this is a a a way to to to give your much more time and limit the number of of CVS you need to patch eventually in the environment. Again this is uh this is not hard science. Okay, because exactly because not you know you it's hard to tell which CVS can be exploited and which can given a given sec profile but as a rule of thumb and many of these compliance frameworks that CISOs are dealing with are working with

a lot of like these high level decisions which are rule of thumb you can save time for yourself in the patching and the number of patching you need to involve >> um when you're profiling all the setcom profiles dynamically and like in the in the cluster. How do you prevent accidentally capturing something like a clone operation or somebody like the danger system calls you wouldn't want but like your container is accidentally you know making those system calls. How do how do you make it more secure than the default one? I'm not really sure that I understand what do you mean by accidentally a and u and I tell you why usually when I get this question I get in this forum please

confirm if I'm right but usually I'm getting this question a way that that people are asking what if I'm capturing during the profiling a real attack there is a real attack and I'm just like capturing I don't know if you saw that I was able to open the shell on the container which was like something strange because maybe I've when I profile this container open the shell and it was just captured and therefore it went into the profile automatically. So yeah, it's it it's something that either you are creating generating these profiles in the CI/CD processes like in in staging environment and capturing and then rolling out to the production with the profile and uh in theory you will

have a uh if you're doing it through GitHubs then then you will have a a clear trail of paper trail of of what what was added and what was changed during the time but I will be happy to discuss it further if if uh we'll have time afterwards. That's it. Yeah. Thank you very much, guys. [Applause] [Music] Happy [Music]

[Music] farmer.

[Music] Hey, hey, hey. Heat. Heat. [Music]

Down. [Music] Down.

[Music]

[Music]

Heat. Heat. [Music] Heat.

[Music] Heat.

Heat. Heat. [Music] Hey, [Music] hey hey. [Applause] [Music] Heat. Heat. [Music] Heat. Heat. Heat.

Heat. Heat. [Music] Heat.

[Music] Heat. Heat. Heat. N. [Music] Heat. Heat. [Music]

[Music]

[Music] Heat. Hey, Heat. [Music]

Wow. [Music] Hi welcome. Thank you so much for coming to Bides Las Vegas. Uh this talk uh Turlo on the side is being given by Adam Adomitis. And I have a few announcements to make before we begin. We'd like to thank our sponsors, especially diamond sponsors Adob Adobe and Aikido and our gold sponsors, Formal and Drop Zone AI. It's their support along with our other sponsors, donors, and volunteers that make this event possible. These talks are being streamed live and as a courtesy to our speakers and audience, we ask that you check to make sure your cell phones are set to silent. If you have a question, please use the audience microphone which is over there

so YouTube can hear you. And as a reminder, besides Las Vegas photo policy prohibits taking pictures without the explicit permission of everyone in the frame. These talks are all being recorded and will be available on YouTube in the future. Uh thank you so much. And with that, >> thank you Dan. Of course, >> good morning everyone. Can we everyone hear me? Good in the back. Awesome. Awesome. No, kind of better. Do you want me to just yell? I can yell for the next hour. Ber. Buler. Okay, there we go. Now we're good. All right. Thank you guys for all coming today. You could be literally anywhere else in the world, but we have all chosen to be here at Vegas to talk

about one of my favorite thacs in the entire world, a group called Tura. This is a group who's known for backdooring the back door. So, fun little fact about me. My name is Danny Adamitis. I am a distinguished engineer at Lumen Technologies. It's a great company that pays me every two weeks. Why I still don't know. Uh I've been in the CTI space for about 10 years. Um I was part of two authorized subscriptions. I drink way too much coffee. So I'm going to be speaking very very quickly for the next 45 minutes. But on the bright side, I do have one redeeming quality. I have this wonderful dog named Cookie. She unfortunately couldn't be here, but she's going to be

with us throughout this entire presentation because sometimes you just need a little bit of levity to cut the undertones of how horrible the internet is. Moving on. So, we're going to talk about how we first discovered a group called Slide Copy, how we then started moving into the Secret Blizzard or Turless Space, and we're going to talk about the larger context, some questions, and what this all means to us. So first section discovering a group called scy copy or as I like to put it the part of the internet that isn't PRC operations ransomware artificial intelligence it does exist people it's just very very small all right side copy for those of you who are unaware of them this was a Middle

Eastern Pakistani based group um they've been active since about 2019 based off some public reporting and their ttps are just a little bit different than the transparent tribe if you're familiar with that group. Um, they're known for predominantly using things like fishing and a combination of open source and custom frameworks. Uh, we're going to kind of get into this one a little bit more. They're kind of just trying to evade EDR. They're not really going to be the most advanced, sophisticated group we've ever talked about before in the world because they are more of a regional actor. They have a regional focus in Southeast Asian organizations. So again, think of places like India. There's a little bit Nepal, Jordan,

Iran, kind of that area of the world. Um they are known for targeting things like government entities, law enforcement, critical infrastructure, military entities, and thus far we have seen a lot of focus on offensive operations for the purposes of espionage. But there was a couple things in there that just made me kind of pause for a little bit that we'll kind of get into in a minute. So one of my favorite questions is whenever I start talking about these wonderful campaigns or one of these actors, I always inevitably get a question from someone afterwards that goes, "Hey, how did this all start?" and like what just tipped you off at first. So, I figured let's just start

incorporating that into the slides. Uh way back in December of I believe it was 2021, we were I was just kind of searching through Virus, as all good people do at 10:00 at night when you can't sleep, working on my Y rules. What else would you do at night? And we came across a wonderful link file that had the words Microsoft 10 ery. Airy was with the I. That's their misspelling, not mine. I know that. That's why I have to call it out. I got yelled at for that one. We started to then analyze this wonderful link file and we discovered this kind of malware family that we're later calling reverse wrap. The really interesting thing about this that piqued

my interest is they had a capability that was specifically targeting removable media and things like USB drives. They were trying to copy all the contents of anything that was on a USB drive and then trying to propagate to it. This was kind of interest. If anyone here has a background in Quo infrastructure, a lot of those networks tend to have an air gap. Oh yeah, but we still need to transfer files from one system to the other. So how do we do that? Propagate over to USB drive. Yeah, there we go. So once we started to extract that, we're able to get some command and control IP addresses. And that's kind of where this all started.

Now the slightly more in-depth version of how that all worked looked a lot more like this. Um, since we do have a full 45 minutes, we'll kind of go over all this. They were sending these zip files as a fishing e lure. Once you unpack that zip file, you would get this Microsoft shortcut file or a link file, LNK. This would deploy some sort of decoy PDF document that would be about some UN sanctions, the state of the military, the latest news sample, just something that we think would kind of peique the interest and make people not realize that there was 14 other things happening in the background. We were then seeing them actually use

things like compromised domains which was kind of interesting because again they were just kind of living somewhat of a way living off of a domain that we think people would actually legitimately visit and it would kind of you know lower that risk score. From there they would then use a whole bunch of executable HTML code which all resides in memory because again writing to disk is very very bad. If you are a red teamr and a new person do not write to disk. They would then start running multiple modules. The first one would check for things like EDR. They would then have another in-memory loader. And for some strange reason, they then deployed two agents on every single device because,

you know, redundancy is important. Don't don't do that either, but it's what actually happened. So, some of the more common malware frameworks that we have observed from this group, uh, we saw a one called Night Fury. So, this was back in 2022. This was kind of uh interesting because it was a new capability that they were developing. So at the time when we did this, it was all written in C++ and they actually had 20 native functions in 20 different spots that they were going to use. However, about 11 of them were just to go into a big loop. So they're doing it guys. They're trying. All right, we're just pushing the prod and they're

just like us. Just like us. Prior to that, they were using another tool called Alor. This was a deli based tool. Um, again, this is on GitHub. It's been around forever. I think it was just way too noisy. Everything was transmitted in the clear. It's just really bad OPSAC. No one likes Delifi. It's a pain. So, they started developing this wonderful Night Fury tool. Uh, the second one and kind of their main kind of flagship module at the time was a tool called Reverse Rat. Reverse RAT was a custom tool written innet. It had all of the fun wonderful features we all know and love. You can upload files. You can download files. You can copy the

clipboard. You can do things like play with removable media. You can write to the registry. You can delete the registry key because you realize that's actually the local one and not the HTML one. It's great. So, that was kind of their main tool. We saw this one. This is where I spent a lot of the time just because it had some of those kind of really unique capabilities that kind of in my mind prefaced that there was this interest in doing more and kind of going after some of these other targets. And then, of course, we found their third tool called Aries. For those of you who know, this is a publicly open-source project. It's on GitHub. It's a Python

based tool. But the fun thing about this one is it runs on Linux because apparently you can also hack Linux devices. No one ever checks those. And again, once we started doing this, we started to kind of write pictures. Um, I work with Ryan. Ryan loves pictures. So therefore, he asked me to always write pictures. And this is what we get now. So I have to explain to him how this all sort of works. Uh we were able to find some of these C2s and then we were able to start enriching this with our net flow. Uh for those of you who don't know, I work for a wonderful company called Lumen. Lumen used to own

level three. Once you own level three, you get to see net flow and things to traverse through the level three network. You get funding sites into the entire world. This is the result of that. So we are able to start identifying some of those lovely C2s that you'll see right there in the center. Once we identified those C2s, we can start to see who was connecting to those C2s over the known C2 ports. This allowed us to identify some things like the Afghanistani government, couple telos in Afghanistan. We also saw a couple other connections from things like the Indian government. We saw I want to say I believe it was two different Indian powers. One of them was

a power generation, one was power distribution and things like the Jordanian government. So if you had a strong interest in the Middle East, seems like a pretty good one-stop shop, right? It's awesome. The other fun thing we were able to do with some of our lovely net flow is not only could we identify who the victims were, we started to kind of do what I call working upstream. So we could see who was trying to remotely administrate all of these known C2s that we were able to find in things like Virus Solo from those lovely Y rules we talked about before that I wrote at 11:30 at night that still somehow kind of work. Not

great, but kind of. So once we were able to look at all of that stuff, we could see them RDPing into this and that allowed us to find that kind of second or bottom set of C2s. We were then able to kind of do this and start to build out that more holistic picture. And once we started to do this, we started to see where else they were coming from Pakistan. If there was any other connections or if I don't know there was any other C2s that were talking to them, that would be so weird. Anyway, let's talk about Hack 5 for a minute. There's no reason at all we're bringing this up. For those of you who don't know, HackF 5

is a commercially available tool. They usually actually have a table out here somewhere. I don't know if they're here or not this year, but the unique thing about them is they provide hardwarebased pen testing tools. So, when I say hardwarebased token, I mean it's actually just think like a physical device that you're going to plug into a USB drive or their very common one if you guys have been around has been the Wi-Fi pineapple where you can use that to kind of break into somebody's Wi-Fi. you have, you know, the was it land turtle, packet squirrel, all of their lovely things. But the really interesting thing that I was trying to emphasize is the fact that you actually

have to be close enough to a network that you can touch a piece of equipment, which is weird because usually we talk about things like remote operations. The fun thing about this is, again, shout out to our oneway hack 5 guys, they have this lovely banner that they display on port, you know, 8080 where it says, I'm a hack 5C2. It makes identification of malicious activity just oh so much fun and easy. And again, once we're able to start searching for that banner, we can start to get a couple of IP addresses. We're calling those our lovely command and control servers. Once we have those lovely command and control servers, do you remember that net flow we talked

about on the PL side? We're just going to throw all those IP addresses in and see who's talking to these things and maybe that can give us some fun insight. And sure enough once we started doing this we found that there was one particular server that was getting a whole bunch of connections from things like Indian governments like Indian national government organizations and a whole bunch of connections from other Indian government organizations that weren't located inside of the area of India probably their ministry of foreign affairs I don't know so this was the actual IP address itself it was the 185 217 and the fun thing about this is we saw remote administration from these other two IP

addresses. Um, for those of you who love reading CTI reports and just love to memorize IP addresses like myself, one of those was actually reported on by team Cry. Shout out to team Cry if you're here as being a known site copy C2. So, we then have a known confirmed C2 with a confirmed malware sample being used to administrate a known hack 5 server and then that hack 5 server going after basically the exact same governments. But the fun thing is with this particular bottom row, it was a hardwarebased solution. How do you get a hardwarebased solution into a government network? >> Would you like to chime in, sir? >> Somebody has to sneak in and put it

there. >> I mean, I don't know. That's just speculation. I'm not here to speculate. I'm just here to report facts. >> All right. Thank you. We'll get you cranks later for lunch. All right. So, this is a this is really interesting to me because I I feel like we always talk about proximity based attacks in the abstract as something that doesn't really occur. And in my mind, this is probably only the third document in extincts of this occurring. Um, the first and the foremost one that everyone usually thinks about was that lovely attack that happened with the GRU in the Netherlands where they were able to see someone using a hack 5 or a pineapple to

try to break into someone's Wi-Fi because they just wanted to do it that way. Um, there was a second report that came out from Valuxy in which there was another Russian GRU group that was basically hacking into the Wi-Fi right next to their target and using that to kind of bridge over the gap. Um, so again, these things do happen, but reporting and documentation of them is incredibly rare. So this is just kind of my other call was just start looking for this stuff because it's probably out there and it's probably happening a lot more than any one of us is actually noticing or talking about. So let's be the change we wish to see in this world.

So after we started doing this, we started looking for other C2s thinking, hey, if they made that mistake once, bet you a dollar they're going to make it again. Maybe we can find some other really interesting command and control nodes that were actually talking to them. And next one we found this first node. This one did not have any banners at all. In fact, there was no IP history about it at all. There was no domains. There was no scan entries. There was no census. There was pretty much nothing in any of the open source which made me go, who are you and why are you here? Enter phase two. Let's go hunting for bears.

So, as we mentioned, the lovely dog. This is Cookie. I'm Booker. All right. Moving on, Secret Blizzard aka Tura. So this is the kind of main premise for the talk is this wonderful group called the FSB Center 16. They have a global remmit and they are I would argue one of the most sophisticated and most advanced threat actors in the world bar and none. Yes, I'm talking to you. I'm even including China in that talk. They are really cool because they like to do things like this attacker in the middle framework where they will actually hijack connections for legitimate things. There was a great report pub by Microsoft I think about a week or two

ago. Um, they also, and this is my fa favorite, started hacking other hackers. And again, who doesn't want to hear a talk about hackers hacking hackers in Vegas, right? They also have that wonderful peer-to-peer framework that used to be what they were calling Snake. And they actually use things like satellite for command and control. And again, they're just, you know, it's just I feel like they take pride in their work. And every now and then, you got to acknowledge the fact that, man, these guys are really prideful and they just try that much harder. Unlike, you know, the side copy guys, they don't really try. They just want to kind of get their job done. in the night school. I guess

it works, but you know, it's not fun for us. We're here to have fun. They are predominantly interested in foreign government entities. So, think of governments, think of Ministry of Foreign Affairs, critical infrastructure, and of course, other hackers that could give them access to everything because that just makes their life easier. And again, they're focused predominantly on long-term cyber espionage. Now, this was one of, I want to say, in my mind, one of the watershed reports. This was done by our lovely counterparts at the UK NCSC. Uh I believe this was about 2019 at this point, but this was the first documented instincts of Terrella actually hacking into an Iranian group. I believe it was called

Hazel Sandstorm. And they were using that to deploy their own tools. Um and again, this was just kind of that weird moment where everyone always talks about, well, why don't we just hack the hackers and just use that to get all the Xfell and do everything we want to do. This was the first time anyone actually documented this in a public space that we can all now talk to and reference. The way they do this is we've observed three predominant tool sets. Um, and this is the part where I will say we did work with Microsoft. We love those guys. If you're watching, they're amazing. Um, they have three predominant tool sets. The first one is called two dash. Two

dash was a super lightweight survey tool. So it's basically saying, am I in a VM? Are you actually running? Is this a real system? can I actually, you know, enumerate things like MAC addresses? And if they said, "Yep, this does look like a real system. We're going to use this to then download an additional module. We're going to decrypt that and we're going to run it all in memory because memory is where you want to live." We then observed a second tool and this is kind of their big one. It's called Tiny Tura. Um, not to be confused with Tura who's the actual group. They also have their own little malware family called Tiny. Um, they were basically used to

again it persists as a service. So again, you do have your persistence. they you've seen them use this for years and years on end. The ability to upload, download code, do remote execution and play with things like the registry. Um, this one is a little bit more fun because the binary actually resides exclusively in the registry itself. And if you've ever tried to search the registry of an entire enterprise is a really really long day for you. So it does work and in there they also then actually have the IP addresses in a different registry key. So that way even if you just recover the binary itself from something like memory, you would then need that second component to

actually know where they're talking to. And the last module we saw was shuzzi. Uh this was used to basically copy the contents of clipboards. Um and again some people say well why would you want to copy the contents of a clipboard? And I look at someone like Wendy and go hi Wendy talk to me about one pass and how everyone just uses a password manager any day and then how you just copy that over and paste it. So if you were to copy the clipboard, you basically just have everyone's, you know, usernames and passwords for pretty much everything. So this is kind of what we saw. This was the initial C2 we were talking about

before up there in the top right hand corner where we saw that starting to communicate with a couple of these Gnome side copy C2s. Um, so again, at this point, we're still trying to enumerate out these side copy C2s through a combination of things like net flow, our lovely Yara rules. Uh we were even doing some fun pivoting based off of things like RDPs and usernames that are being used across multiple things. It's super fun. And we noticed that at a certain point, I think this was around May of 2023, we saw that there was actually getting beaconings to that Turles C2 from a Afghan government IP address. Um, so this was kind of a very unique and

interesting thing where you know before they were hacking the hackers. Now they're hacking the hackers to use their C2 to deploy their own malware to a particular network. Why they're deploying their own network. Maybe they just didn't like the way they were doing things. Maybe they had an interest in a different part of the network. Maybe they wanted different files than you know the Pakistanis were grabbing. There is a number of reasons but again we believe that the initial access vector was actually someone else's command and control. We then actually saw them starting to poke at some of the other C2s and we even saw them starting to poke at the Pakistani ISI operators themselves. Now

again, I know that there's probably going to be one person here who's going to scream pcap or it didn't happen. So this is my pcap or it didn't happen slide. So this was the known site copy is or I want to say nongside copy IP address. This was actually based in Pakistan. um it does resolve to a real IP address and then we actually saw this being used to RDP into a known site copy C2. This was the 130 185. Um and the fun thing about this is you can actually see the first scene and last scene from our lovely telemetry at the exact same time. You'll notice the exact same IP address was also talking to a turtle C2 over its

known 9443 port. And the fun thing is if you look at the first scene and last scene times, it's the exact same time. So again, this is the part where I know someone's going to inevitably say, "But it's a dynamic IP address. There could be multiple things happening. How do you know it's really the operators who are RDPing while Terrell is hacking into them at the exact same minute?" This is the part where I'm going to go, this is about as good as it's probably going to get for us. So yeah, yeah, I feel pretty good about that assessment that, you know, they actually did manage to move from the command and control servers of

the site copy actors back down to the actual operator's workstation. This was again a huge thing in my mind. We've never actually had a documented instincts of a known nation state threat actor compromising another known nation state threat actors operating workstation. So of course I then called my spirit animal cookie and said, "Hey, what do you think of all this?" And she said, "What the duck?" So this is kind of what it looked like. Um, again, we were able to then kind of see them continuously interacting with a whole bunch of these Katabo IP addresses and Nexiums. We were able to see them continuously interact with the Afghan government for about months on end. I

think we still saw stuff as recently as you know late 2024, early 2025. And then sporadically we would actually see some of these Pakistani IP addresses popping up in our telemetry where they were actually interacting with the exact same tur IP address that was being used to interact with all the other C2s. Yeah. Yeah, it's fun. So section three, the absence of data or when not having any data becomes your main data point. So as we kind of talked about before, we are talking about some of the main malware families that we were predominantly being seen from the side copy thread actors. Um so again we have a lovely report you guys can read it. The main ones were there was a tool

called Weiss Cotton. This was a go-based tool. We think that they were using this predominantly to go after some some of the like more Linux Unixie systems because you know you get that lovely portability that everyone loves. Um action rat this was kind of like a variant of reverse rat. Um this was kind of a little bit different but again x.net it's targeting things like Windows systems. We saw crimson rat. So this was actually being used to target things like Android devices. Um again another super well-known documented family. We observe things like Alor again super well-known documented family. It was written in Deli. It's being used by everyone and we actually saw the turtle actors interacting with almost all of

them except for the Aries rat. And this was the part where I just couldn't quite put my finger on why would you hack into seemingly everything? You were on the operator's workstation. Clearly you have the credentials at that point in time. Why are you not targeting this one particular group? So if anyone here was at Pivixon last year, we had our older counterpart Seth gave a wonderful presentation about Aries rat and how he was tracking that. Shout out to Seth. Um, and we were able to start kind of going into this and this is where we started I kind of jumped down this rabbit hole of why is what is so different about this one

particular malware family than all the other ones and why are they not targeting them. Um, and this was and again we so we started kind of enumerating out all of their infrastructure. We did that same methodology we talked about in the earlier slides where we were able to kind of see where they were coming from in Pakistan. We were looking for remote connections. We were then able to actually start seeing some of these IP addresses and we saw what looked like, you know, sustained connections to their admin page. Um, fun thing about their admin page, they decided to try to hide themselves as a PFS firewall in a VPS because that totally happens in the

normal world all the time. And while that might help them evade scanners, if you have something like Neflow, it just stinks like a sore thumb that all of a sudden you're going to see 10,000 connections to some random ephemeral pore and this one random thing. And then if you actually do something like URL scan and you see PFS, it it becomes pretty easy and I just don't quite understand why they never targeted this. And then that kind of led me to my next kind of correlation. So we saw our wonderful secret blizzard actor or Tura in the top right hand corner. We saw them hacking into all the C2s. We saw them actually deploying their stuff to places like the Afghan

government. We actually saw them in the Pakistani network. But for some reason, we never actually saw them using that same C2 to deploy any of their malware into places like the Indian-based networks. And this was just kind of a really in my mind interesting data point of why would they go through all this trouble where clearly they have the capability, they have the access, they've done this before. what makes some of these other Indian government targets so different that they don't want to actually deploy any of their own malware. So this is so again kind of going over some of this. We saw them in my mind they clearly had the capability. They had the

capacity to do this stuff but for some reason they didn't. Um so this is kind of when I put on my lovely international relations hat and we started to look at this from a fun geopolitical lens and wondering about the IR and the connections of all of them. Um the only thing I could think of is there was a geopolitical preference to not be discovered in any of these networks. Um the fun unfortunate thing about performing these works of operations is anyone who does them know at some point in time you're going to get caught. It might not happen today. It might not happen tomorrow. It could be a decade from now. But every single person who

has done any sort of hacking event at some point in time runs the risk of getting caught. And if you get caught it's going to cause a bit of a stir. So there is a fun connection between the Indian government and the Russian government. um as we kind of well I think I mentioned this before Modi was recently reelected in 2024 and his first international visit was to Moscow. Um so again that's usually where you signal who your strongest preference is or where you're trying to build relations. Um so again I think that there is a whole kind of budding relationship there. Um there's also some connections where the Russian government's selling a bunch of their oil over to the Indian

government and again they need to continue to sell that oil to have funds to then prepare for the war that's still going on in Ukraine. Um there is also this fun thing where a whole bunch of Indians have been being duped into you know being conscripted into the Russian army. So again there might be something where you know they're clearly benefiting from this and you know they really like the status quo. They might still have kind of what we call this information need or this desire to learn about what's happening in India because it is a large regional player but they don't might have the political will to actually get caught doing this. So what

do you do when you need to have information about them but you can't actually be in the network yourself? This is that fun little middle ground where you're like, "I'm not touching you. I'm not touching you. I'm I'm not not I'm not I'm not not touching you, but you're still basically doing exactly that. You're just kind of poking the bear. You're collecting all of that wonderful information you need and you can use that to still kind of answer all the questions that you need to do." Um, and again, this is not my opinions. This is European Council for Foreign Relations and BBC. So, yeah, good enough for me. Section four, conclusions. So, this was kind of my favorite topic was what

happens when you start backdooring other people's back doors. Um, in my mind, there has been five documented cases of Turtle doing this. And to my knowledge, I think they're probably one of the only group that's been publicly attributed to performing this actual trade craft. Um, this is kind of, you know, sometimes called a stunt hack or, you know, something where everyone talks about how great it would be if they could do this. Um, this is one of the few groups that's actually putting that into practice. Uh the first one was of course the NCSC thing. There was a second instincts um and this was documented by Mandant and now Google where they purchased an expired domain and they used that to

drop their own tools. Um we've also seen this from secure list where there was a CIS actor who was using Thomas and then they were then dropping their own tools. This is of course the fourth instincts and of course we saw the fifth one which was Amade. Um, if you guys are here and from the cyber crowd crew, you know that this is a well-known malware. It's a platform service. It's being used by cyber criminals. It's seemingly everywhere. And they were using this to get more information about people, particularly in Ukraine. So, and kind of the last couple things, why should I care about any of this? Well, I feel like there's been, as we

kind of talked about, a number of cases where we've kind of seen what I call snapshots in time. No one has ever actually done a two-year comprehensive study of what exactly is happening and how they move from one C2 to another back down to the workstation using that to then deploy their own tools. Choosing to go into some networks while not choosing to go on the other networks. Um I feel like this kind of really helps reveal a lot about like how they think through these problems and how they work through them in a way that you know you're just never going to get if you just did a lovely IR and you go, "Oh,

cool. We found two malware families on the exact same system at the exact same time. That's so weird." it it doesn't really talk about how they view these problems and how they start to think through them and where they decide to go. Um, the other fun thing that this is one of my favorite data points, not only did we see them observe or hack one C2, we observe what looked like signs of compromise or beaconing from 33 different C2s over two years. Um, so again, everyone kind of goes, "Oh yeah, well we think this happens and they probably have access to a bunch of things." This is one of the few examples where we actually have what I call hard

data points to point to and talk about how exactly this is occurring. And again, in my mind, the greatest thing is this is the first time we've actually seen a nation state move back down onto those operator workstations. And at least in my mind, if we know if it's tiny tura, they had persistence, they had the ability to move laterally. If anyone here works in the red team, how many of you guys have EDR expired on your home workstation where you do all of your operations? And how many of you guys start checking that registry for weird keys? Yeah. Yeah. So, if you were hypothetically on that network, they probably could have persisted there for

a very long time and gotten just absolutely everything that they wanted. All right. Some of the fun unsolved questions, as any good person does, you always talk about the unknowns, unknowns, or the things that you weren't quite able to figure out, but maybe some of you guys do. So, if you do, come find me later. I'll buy you a beer. All right. So my first thing was how was site copy able to you know get some of this hack 5 equipment in. Um this is the part where it would indicate that they either hacked some level of access to these facilities. They kind of did something where they were hacking from the parking lot or they had someone on

the inside. But that's just kind of my big thing was I think it's not often that we actually get to see these hack five devices being used in the wild for real world attacks. And then the question always becomes how are they doing this and do they have some sort of travel team very similar to what we saw from other operators like the GRU. As for deter questions, why did they choose site copy? There's again seemingly every nation in the world has their own hacking organization. There's how many private sector offenses? There's how many cyber criminal groups? There's how many botniks. They could literally target anything. These guys are actually really really good at what

they do. But what was it about this one particular group that made them stop and go, "Yes, that's exactly who I want to start targeting and that's why." Um, the other big question that I don't think we're ever really going to get an answer to was why were they so comfortable doing things like deploying their own malware families and their own tool sets in places like the Afghanistan government, but they seemingly chose not to do that for places like India or Jordan or Iran or some of these other networks where we actually know that the site copy actor had access, but we didn't see anything from the actual Trella side of things. Um, and again,

this was the other fun thing is if we're going full circle back to the very beginning, and the reason this first caught my attention, there was this weird angle where we saw them targeting what looked like power generation and power distribution companies. We know that they had things like USB modules. We know that they had the ability to do things like copy clipboards. We know that they were trying to propagate and we believe that the Pakistanis were just doing that for their own purpose. Um, Secret Blizzard also has their own capabilities to go after some of these greatful infrastructures. What happens if you're trying to then do an IR where you see Turbo walking in with stolen

credentials being used by a different actor into that network? This is kind of this fun thing where I know this is where everyone always says and clamors that they want immediate attribution from doing some of these instant responses and they want to know exactly how this happened and what happened. This was something that took me over two years to piece apart. So again, if you are one of those directors who's demanding for things, just give your people a little bit more grace and mercy. Sometimes it takes more than five minutes to kind of put all this together. Okay. And at the end um this was actually a joint endeavor as I mentioned before with our lovely counterparts at

Microsoft. There is a great group of guys out there who was focusing on the Russia group. Um they also did their own blogs. They kind of focus a little bit more in the malware. Um that was called operation freeloader and then they did the second one on Amadeay. Um we at Lumen kind of focus a lot more at network telemetry because that's kind of our special sauce. Obviously, they're very good at this. And again, better together. Everyone wings except for the Russians, but we don't care about them. Okay. And questions. Sorry, I'm a little early. [Applause] Do you want me

I think I was short by a couple minutes. >> Yeah, >> thank you. >> Thank you for the great talk. I really enjoyed it. Uh I had the question that you mentioned you saw some evidence of this kind of activity by Turla into possibly 33 33 different C2s of different malware families. So maybe I missed this but some of these were the ones used by the uh Pakistani ISI. Yep. >> Some of these were maybe the cyber criminal botnetss etc. >> So it was 33 of the Pakistani ISIS and then there was also cyber criminals over here too. But I had to kind of time scope it a little bit otherwise I would just keep doing this forever.

>> So within the 33 that you mentioned all of those were uh Pakist copies infrastructure. >> We assessed that all of them were associated with either side copy andor transparent tribe. Uh so it's two different Pakistani based groups. >> Okay. Uh thank you and follow-up question. I didn't really totally understand your last comment about why are there get why would the ISI get into power generation power distribution in India? It seems kind of obvious to me. So what are you saying is why is Turla getting into there as well? I didn't quite understand that. Thank you. >> Yep. So we it does make sense that you know the Pakistanis are very interested in India. There's been a a bit of a warm

hot cold conflict there in the Kashism region forever. I'm sure you guys are familiar with this one. Um I I think it was more of the point I was trying to emphasize was that while the Pakistanis appear to be kind of prepositioning themselves, the way you do that is obviously gaining some level access, getting things like admin credentials, getting things like network maps, that's exactly everything that Turbo would need if they wanted to try to then move into that same network. So again, it's one of those things where what happens when you have I want to say side copy performing all of your kind of initial reconnaissance and you're laying out the land and then you have someone like Tura

coming in on top of that to actually then start hacking that and if you're someone who you know works for an instant response firm and you're trying to piece all that together, it's going to look very weird and very awkward for you. So again, it's just kind of in my mind trying to bring up the concept that if you see signs of compromise, that means that there could actually be two or three other actors who are also residing in that same network who have used the same trade craft. I hope that makes more sense. If not, I'll find you afterwards. >> Excellent. Now, great talk. I'm curious uh is there any way you're going through

differentiating between tur purchasing access versus turer compromising uh groups like side copy and uh and even more broadly across the community. I mean what are some of the indicators around your analysis of how you differentiate between those two? Thank you. >> Yep. So for things like this one um in my mind a little bit easier I we there's been no evidence or documentation that people like copious ever sold access to anyone. they appear to only be hacking for the purposes of their own geopolitical interest. Um, we've also kind of seen things before with I think some of the Amaday where typically if you purchase access you'll only get access to like a set of Craigslist or

you might get some information. Um, I think we just saw what looked like just a large amount of data expo from some of the backends that again we only ever see the actual cyber criminals coming in from. So the fact that they're kind of using next same door in my mind indicates that they didn't actually just purchase something. They probably, you know, gained some sort of access. And again, as I mentioned before, these guys are really savvy. They're good at JavaScript. They're good at all this. They probably found some sort of weird, you know, off bypass that. Again, I'm sure you're going to hear all about off bypasses with things like cold book striking, stuff like that. They probably

have someone doing something similar. But great question. Anyone else? I'll also be around for, you know, parts of the day, too. So, if you guys have something that you don't want to put on camera, I'm around. Cool. Awesome. Thank you guys.

[Music]

[Music] Heat. Heat. N. [Music] Heat. Heat. [Music]