These Are NOT the Vulnerabilities You Are Looking For: Hiding Vulnerabilities in Containers

Awesome. We have a lot of people. Now, this talk is supposed to be a little bit interactive. So, if I talk, this isn't going to be a talk. It's going to be a monologue. And those are not fun. So, let's make it a little bit fun because the whole point of doing this is to make it useful to you. So, I got a question for you. How many of you have used any kind of security scanners? Right, cool. How many of you used container security scanners? Okay, a few. Good. We'll be able to work with that. What kind of scanners have you used? Can you name a few? >> Okay, that's Palo Alto, right? >> Anything else?

>> Algri open source. This is a popular scanner. Anybody use Docker Scout Snig? Okay good. Close enough. Close enough. Now, if you used any of the security scanners, you can relate to the next slide.

You know that feeling when you run your scanner and you get a a Christmas tree report? So, what is it? it it's a whole bunch of findings, tons and tons of findings and you don't know what to do with them and you know most of them are useless, right? So this is about something like that where I'm going to have a demo app with lots of findings and no time to fix them. So we'll find a way to make those vulnerabilities disappear. And you mentioned gripe. This is actually me using gripe. And this is uh the demo the containerized demo app. And in the talk summary, it mentions 5,000 vulnerabilities. But in reality at this point, cuz I ran

it earlier today, it's about 9,000 vulnerabilities. Can you imagine how many vulnerabilities it is? I actually wanted to bring um a few books with me just to show, but it ended up being a stack like this. Like literally, I'm not kidding. And I just couldn't fit it all in my backpack, and it was just too heavy. So, it's just impossible to fix all that. Obviously, you don't get 9,000 vulnerabilities all the time. But still, even with a few dozens, you always wish if you could just make them disappear. Now, I'm going to try to run a few scans just to see just to show that it's not made up. All right. Local host. Not that one.

Okay. We got a few. All right. >> Oh, yeah. Exactly. So, I'm gonna go ahead and copy

I'm going to run it live so you know it's real. So it's not fake. It's not AI generated. And by the way, if you expected to see any kind of AI, no AI, AI free

and All right.

And I'm going to kick off a few more scans too. Um, okay. I'm also going to run a Tribute scan. Anybody use Tribute? Okay, good. So, we're not going to get 9,000 findings here, but we're going to get a few. And we'll run a sneak scan as well. Not Not going to be 9,000. still going to be quite a few. We'll run Docker Scout

and I don't know if anybody used OC scanner from Google, but we'll run it anyways.

And this isn't a scanner. Sift scan.

Okay. Besides,

this isn't a scanner. This is an sbomb generator. And if you're dealing with security, you're probably using some kind of asbomb generation tool. So if we go back and look at the results, lots of them. Yeah. 9,000. Now what if

we we can make them all disappear?

Ever wanted to do something like this when they said, "Hey, these are not the droids you're looking for." And we're going to try to pull it off. Something like this. But in this case, instead of Star Troopers, we'll have vulnerability scanners. And we're going to tell them, "These are not the vulnerabilities you're looking for. So for this, we're going to run a tool that I built, but you can do it manually. It's just a a way to accelerate the process and it's called Min toolkit. It used to be called Docker Slim and I'm the Docker Slim guy. And it was created to slim containers, but it has an ability to hide vulnerabilities, too. Guess why I added that capability there.

All right, I'm going to pick a screen.

So in this case the tool is OB1 and it's doing a little bit of magic and again it's live. Just wanted to make sure that you see that it's not fake, it's real and it's it's looking at the container image. It's doing dynamic and static analysis and then it reassembles the container. And because I passed a few flags to obuscate the data in the container in the reassembled container, it also removes additional things that the vulnerability scanners need to find uh the vulnerabilities. And I have a question for you. What kind of information do vulnerability scanners need? container vulnerability scanners need. >> Any guesses? >> That's true for the applications you usually, for example, the demo app is a

node app. It has a package lock and many vulnerability scanners, they look for the lock files and they get the package information from it. And it's similar with the um operating system packages as well. Uh there there's metadata OS package metadata that you need. But the most fundamental piece of data is actually the operating system uh version and the distribution name. And if you remove that metadata which is located in um in a file in etc-re, you're going to break a lot of vulnerability scanners just by doing this. But let's see what happens. So we got um we got an image obuscated image. I know it's tricky to see, but I'm going to find the name.

It's in there. All right. Completed output ID right there. Here's the name. It's the same image name, but it obuscated uh tag name. So, let's try to scan this image. with with grape.

And let's see what happens.

What's going on? What just happened? There are no vulnerabilities. We went from 9,000 to zero. So, let's take a look at the magic and let's explore the app. So the app itself is a simple Node app, NodeJS app in a Debian container. It has a few endpoints. It has a few vulnerabilities in it, including this old CVE with a command injection. Everybody knows command injections. If I have time, I'll actually exploit this command injection in the obuscated image. Even though it says there are no vulnerabilities, but they're actually there.

So if we look at the image and we're looking at the um original image and if we go to etc OS release is actually a sim link to user labs

right there. And this is the information that we have. Not a lot. But let's take a look at the image that's obfuscated and it's that that obfuscated version. Let's take a look at it directory. There's no OS release file. And if we look at user lib also no OS release. So this is one of the first things that happens. Just remove the OS industry information and for example the uh the Google scanner OSD scanner. It's dead dead in the water and it can't find anything else. You don't need to remove anything else. But what else is missing? So we just mentioned a few minutes ago that we need OS log data and also similar data for the operating system.

And what is that data? Now if we look

if we look in the var directory. I'm going to hold it. Var lipkg

status. This is actually where the metadata for the OS packages deboss packages lives. Guess what's missing in the offiscated version? This part is missing.

But let's take a look at the application itself. So the application if we're looking at the obuscated image

uh the application is in the opt demo service directory.

There's a couple of strange things there. I don't know if you know much about node applications but normally you don't see those sim links but those sim links is just an additional layer of opuscation for the current versions of the vulnerability scanners they don't do much but still it's there but if we take a look at the package json well first of all the lock file is missing and if we look at the JSON uh package JSON um file. We'll see a few things there.

We'll see the names of the dependencies we have in the application, but the versions are a little strange. Okay. So, what the tool did, and I'm going to go back to the slides.

It removed the OS metadata, the DRO information, the OS package information and it mutated and removed application package information. So the scanners are now blind, but the app is actually functional. And if we go

All right. So,

so if we run the app,

it's actually going to run.

And if we go here,

we're going to go to ping. And for some reason, it doesn't want to Oh, right there. So, it is running. But I promised. Yeah. So, by default, it's gonna ping the Google domain.

It's a little slow. The network is slow.

All right. And this is the actual exploit that I wanted to show. It's the command injection. All right.

Docker ps docker exec.

And if we go. All right. It should be being attacked too out

and the vulnerability is still there. So, we made the vulnerabilities disappear, but they're still there. You can exploit them. And guess what attackers can do with something like that? I'm pretty sure a lot of you or your companies use third party images. And if somebody hides the vulnerabilities, you run your vulnerability scanner. It tells you it's okay. You put that container in your environment and now attackers can compromise your environment anytime they want and they don't need to persist access. Fun

>> one minute. So you could do one quick question. One quick question. >> Is there a way to avoid vulnerabilain?

>> Yep. you can avoid some of the vulnerabilities and uh the tool that I created docker slim is actually one of the options where it can remove the packages you don't need or you use distro but still this application is in uh this vulnerability is in the application itself you can't remove the code that you need in the application so you can remove some go ahead to stop my developer from doing that. So I don't they don't get my ticket to fix their package. >> That's a good uh conversation with your developers and we can talk after the talk to discuss the details.