Navigating the Virtualization Battlefield: A Deep Dive into Hypervisor Attack Vectors

around for a long time, long before large language models and cloud. In fact, those technologies run on top of what we're going to be talking about, which is a hypervisor. So, a hypervisor is a system that's used to run virtual machines. There are different types of hypervisors literally. There are type one hypervisors and there are type two hypervisors. Many of you might use type two hypervisors like virtual box or VMware Workstation to run VMs in your Windows or Linux systems. We're not going to be talking about type two hypervisors. We're going to be talking about type one which is essentially a bare metal system that is a server and its job really is to just run virtual

machine workloads and orchestrate those workloads. And we're going to really focus in on a particular type of hypervisor called a VMware EXI hypervisor. And the reason for that is despite the fact that this product, this technology has been around for many, many years, it's really only the last three or four that the number of attacks against it have skyrocketed. Many, many attacks against these systems. So, who am I? I'm Austin Gad. I'm the CTO and co-founder of a company called Valley Cyber. Prior to starting the company, I did a master's thesis on automatic exploit generation. So, I built a system called Rage that focused on attacking Linux systems. And Rage would bypass modern defenses like data execution

prevention, ASLR. It would automatically do memory corruption exploits. And so a lot of focus on attacking Linux systems. After that, I spent some time in the Air Force. And that's because for my undergrad, I was at the Air Force Academy. And so as [clears throat] part of being at the service academy, you spent some time active duty in the Air Force. And I was responsible for security in the nation's satellite infrastructure, particularly from a software perspective. So I was doing a lot of software security for satellites in orbit, satellite ground systems and it turns out those systems all run Linux operating systems. So I was doing a lot of defensive Linux security once again and that's really what motivated me to

start Valley Cyber initially is my experience at that time is there were a lot of challenges with the Linux security tools that we had. As Valley Cyber grew, we started to focus in on this area of hypervisor security. So we're really focused on hypervisor security today as a company. So that's some background and let's get into the attacks. So hypervisor attacks have been increasing at a massive rate over the past few years and there are a few motivators for threat actors that are going after hypervisors. One of them and really the most common one is ransomware. So the idea is the attacker is going to get into the hypervisor and they are going to encrypt all of an

organization's virtual machines all at one time. And so you can imagine an organization has their private cloud running on ESXi. all their VMs get encrypted all at once. It's a very, very bad day. It's a very devastating attack. They're going to be very motivated to pay a ransom from that attack. And so, some great examples of this, MGM Resorts was one of the first big ones. That was about $100 million worth of damage from the attack itself. Then there was a $40 million class action lawsuit or $45 million. That was just the settlement, too. So, that doesn't account for all the legal fees that MGM had to pay. So these attacks are not only expensive

from an operational standpoint, just getting back online, they can be very expensive from a data theft standpoint cuz a lot of these ransomware attacks will leak the VMDKs for the virtual machines before they actually encrypt them and then you have to deal with lawyers coming after you go in for class action lawsuits. Now there's another type of attack that is very popular on ESXi systems and it's less common, less known, but it is publicized. And so MITER was breached in their VMR environment in May of 2024. And this attack was a little bit more sophisticated. This was a nation state threat actor. And the goal of this threat actor was to dwell on the hypervisor, gain persistence there, and

launch further attacks into the network from this very privileged point. And so the idea is there's not really a lot of detection response happening on hypervisors themselves. People don't really run EDR on them. And so if you are on a hypervisor, you can sit there for a long time without being noticed by any sort of defensive technology. And it also gives you a lot of power. So you can run virtual machines, you can do something called a rogue VM, which we'll talk about in the presentation later on. And you can steal VMDKs, you can do a lot of nasty stuff from within the hypervisor. So it's a very nice position for the attacker to get onto on the

network. And about a year after MITER's breach, they they did some research and they released an update to the MITER attack framework that we'll talk about. But a common misconception I I run into is that people think VMware is dead. People think VMware is dying because Broadcom acquired VMware. The Broadcom changed the pricing structure of VMware quite a bit. So, a lot of organizations are moving off of VMware. And it turns out this isn't actually the case. The market is still heavily dominated by VMware. In their most recent earnings call, I think Broadcom said that about 87% of their top 10,000 customers are remaining with VMware. So, what you're typically going to see are smaller

organizations might be moving off to Proxmox or other nutanics, other sorts of virtualization solutions, but large enterprises are largely staying with VMware. And so, VMware is definitely here to stay. And another motivator for attackers to go over to go after ESXi instead of Nanix or Proxmbox is the idea that VMware is the most prevalent hypervisor. So if I go after it, if I write my malware for VMware ESXi, I'm going to get the more more bang for my buck. It's going to work better for more targets and more systems I'm going after. Okay, so there a couple different ways these systems tend to get attacked. One of them is going to be credential compromise which we'll talk about where

an admin gets fished and their credentials are stolen that allows the attacker to log into the VMR environment. There misconfigurations too where you'll have assets exposed on networks that they shouldn't be exposed on and then just vulnerabilities, exploits and CVEs. And so we'll talk about all this stuff. So minor attack 17 as I mentioned it came out just in April of this year and it did something pretty interesting. So before you had three different platforms in the MITER attack framework, you had Windows, Linux, Mac, the big operating systems. They added ESXi as a new platform. And they did this because they're seeing so much threat activity on ESXi systems. So it's definitely a type of operating system

that is now something you really need to consider from a security perspective. And they added new TTPs specifically for ESXi. And so we'll talk about some of these TTPs as we go through things. So the first attack we're going to talk about is really going to go back to the start when these ransomware attacks really started to get popular. So this was in early 2023 and it was an attack called ESXir. And so ESXir used a CVE that was about 2 years old at this point from 2021. And the idea was I'm going to go after all these ESXi hosts that are publicly accessible on the internet because people don't have good firewall hygiene. They're exposing assets that

should not be exposed and they might be vulnerable to the CVE. The CVE is very nice for this type of attack because it is a remote code execution exploit. So the idea is the attacker doesn't need passwords. They don't need anything special. They just need to see that ESXi box on the network. They can attack it. They can gain administrative privileges for it and they can launch code and malicious ransomware from that vantage point. And so the the final technique that we're going to see is technique 1673 which is virtual machine discovery. And so something that attackers love to do when they launch ransomware is first understand what all the virtual machines are because they have to shut them off.

And they have to shut them off because the VMDK files that are used by virtual machines while they're running on ESXi, they're locked. So the goal of the attacker is to really get after those VMDKs, that's the virtual hard disk. They'll try to export them. They'll try to steal data from them, but they'll also try to encrypt them. And if the files locked, they can't do that. So you kill the VMs, you find the VMs, you kill the VMs, and then you do your ransomware from there. So we're going to use a P that was available on GitHub. It's still available on GitHub. I wouldn't say it's a weaponized version of the exploit. It does take a little bit of time to run,

and we'll talk about why that is as the exploit runs. So now let's see a demo. So, we're going to go after a VMware 6.7 system that's vulnerable to the issue. And this system has a few different VMs on it, Gandalf, Froto, and Gimy. And the goal of the attacker is to encrypt those VMs. And so, on the right side of the screen, we have the PC that we're running. The left side of the screen, we have a netcat listener. So what the exploit is going to do is it's going to connect that to that netcat listener there on the left and it's going to allow the attacker to run this uname a command that has been queued up. And so

the attack spins up. Essentially this attack is a two-stage exploit. It is a buffer overread followed by a buffer overflow. So the first goal of the attacker is to leak information out of the address space for the vulnerable SLP process and understand the location of code because they want to run their own code. they want to run this command which is this make nod tmp backpipe command. This will ultimately allow them to talk back to that netcat session that we set up in the other terminal. And so the attacker is first going to do something called heap grooming where they set up the objects and the structures on the heap of the SLP process so that they're arranged in a

way that when they do memory corruption they're going to get the data that they want. So the first goal is to leak data out of the process and to understand the address of specific functions and specific code that we want to execute. So we're going to look for the code addresses of functions like system that allow us to run system commands that will ultimately allow us to run that make nod tmp backpipe command. And so once the exploit gets to this point, we can see we've leaked a bunch of stuff. We've leaked the system address, environment address, free hook address. All this stuff is very useful for launching the next stage of the exploit which is the arbitrary code execution

itself. And so this will go a little bit faster. We'll go ahead and send a bunch of packets and eventually we will pop a shell over here on the left side of the screen. So thatame- hit command runs. We can see that we're now running commands on the ESXi67 system. Cool. So I'm going to pause the video here at a key point. the attacker is going to do first is they're going to run wget to download and then run some malware. And once they run that command, script is going to output some stuff. And I think we we missed it by a little bit. So I'll jump back real quick. See if we can catch it.

All right, there we go. Okay, so first prints out kill VMX. And so this is what I was talking about. This is that virtual machine enumeration and that goal of killing the VMs before we actually run the ransomware. That's because those VMDK files are locked while the VMs are running. So, we have to shut down the VMs. Then, we run our ransomware. Okay, cool. So, it runs its ransomware, goes through, it encrypts the files. If we do an ls on the data store, which is where the VMDK files are located, we're going to see the Gimly VM now has an arg extension. So that's because it's been encrypted by the ransomware. The ransomware gives it

therx file extension. And if we try to spin up this VM and go back to the web console for the ESXi system, we'll go ahead and try to run it. We get this nasty red error saying the VMDK file, the VMX file cannot be found. And so this is a bad day if you're system administrator, none of your VMs are running, and you're sort of in panic mode. This is not a good situation to be in. And just to give you an understanding of the scale of this attack, there were hundreds of organizations that experienced this all at once. So at the time that this attack was really launched, a rapid 7 did an analysis of just publicly accessible

systems on the internet and they found that there were about [snorts] 19,000 that were present on the public internet that had this vulnerability. So they were able to be attacked by this exploit. So that's hundreds of organizations that were impacted all at once. And the good news is that because this was so impactful, because so many organizations were impacted, a lot of organizations recognized, okay, we can't put our VMware assets on the public internet. That's bad security practice. We're not going to do it. That's going to make us more secure. We're not going to be vulnerable to this sort of thing. Except they didn't. And if you go on Showdan today, you're going to see

there's about 20,000 systems that are ESXi systems, publicly accessible on the internet. And some of them have been encrypted. We'll see that a little bit later. But the the key thing here is you definitely want to make sure you're firewalling off these systems. In particular, the SLP process has had a lot of CDEs against it. A lot of organizations don't really need to use it. There's a firewall rule you can change on your ESXi hosts to specifically disable the SLP service access. So, that's something I would recommend turning off if you don't absolutely need it for some reason that I I don't understand because like I said, not a lot of people need this service. And obviously, you want to

patch. you want to do network segmentation with your firewall and virtual patching is also a very interesting concept that you can use here. So virtual patching you can think of of it kind of like exploit prevention. So one of the challenges with hypervisors is because it's core infrastructure you have VMs running on it. Some organizations can't actually patch them. So think about critical infrastructure maybe it's a natural gas company or something. They have these VMs running on these ESXi systems. They're running critical workloads and they just can't shut them down. And patching them does require a reboot. So you can't have that downtime. And so what virtual patching is is it's this idea where we're going to use a firewall

or some other security tool that's going to look for the exploit behavior itself and it's going to block the exploit behavior. So we're not actually patching the code. We're not actually fixing the vulnerability. We're just making it difficult or perhaps impossible for the exploit to be successful. >> Can I just ask um did I read this right? When you ran your payload and you exported your shell, you using netcat. >> Yeah, it was a netcat listener there on the on the left. Yeah, >> we had a netcat listener, but when you exported the shell on ESX, >> yeah, netcat is available on ESXi by default. >> Netcad is >> it's installed by default. >> Installed on ESXi.

>> That's right. Yeah. And there's a lot of other stuff installed on ESXi that's not so good, which we'll get to as as things go on. And and really uh it's Bizzy Box that's installed on ESXi. And busy box has a bunch of different commands it can do under the hood. So I I think the next attack that we're going to talk about shows us that firewalls aren't enough. I think for this first one, a firewall really is enough. If you just block that port, you're not going to be in trouble. If you take your systems off the public internet, you're not going to be in trouble. But attackers know how to get around this. I mean, this is a security

methodology from decades ago where we're going to have a really strong perimeter. We're going to have a really strong moat. And then the inside can be soft and gooey, but that's okay because we have this big moat around our infrastructure. But we know this doesn't work because attackctors just find a way through that moat usually the same way that the actual administrators for the system get through that moat to do their work. And so that's what we're going to see with the MGM breach. And so the MGM breach was an example of lateral compromise where the attackers used some fishing some sophisticated fishing techniques. I think scattered spider is responsible for this particular attack and they've been going off the rails

this year with ESXi attacks. There were a whole bunch by Scattered Spider that were listed on one of those earlier slides. And the way they typically like to go after organizations is to first target the help desk. So they'll target the help desk and they will use things like SIM swapping where they fake a phone number. They can also fake voices using AI to make it sound like they're calling in to someone else. And you can imagine these poor IT admins that are running these help desks, whether they're internal help desks or a third party help desk is often what they target as well. They're dealing with dozens maybe hundreds of calls a day. And

they trust most of these calls. I mean, you often if the phone number is right, if the voice is right, you're just going to be very trusty in that situation if you're dealing with call after call after call after call. And so what the attackers did is they called in with a fake phone number and they said, "Hey, we have I I lost my password. I need a password reset." They received the password reset. That gave them access to the cloud environment. And so they did some lateral movement in the cloud environment. They gained access to the VMware environment. Now, this part was not really published. They didn't really talk about all the details of what they

did when they're in the VMware environment, but I'm going to sort of expose that for you because this is typically how these attacks work when the attacker gets into the VMware environment itself. So what the attacker likes to do is they first target something called Vsenter. So Vssenter is the management console for all the underlying ESXi hosts and Vsenter allows you to do things like turn on services or disable services on the ESXi systems. Notably it allows us to turn on SSH. And so SSH can be used to log into the systems and run commands just like SSH on any other box. And this is what attackers love to do. They're just going to use SSH. They're going to use the

normal tools that administrators normally use because those are normally allowed. They're not abnormal. It's normal behavior that we're doing to get through that tough perimeter and go after the gooey center of the VMware environment. And so we'll see this example here. We'll sort of assume that we've stolen the password for this vssenter system. We're going to go after one of these ESXi hosts and we're going to turn on the SSH service for it. So we'll go over to the configure tab. You can script all of this, by the way, but we're just doing it by the guey here. And we're going to go to the SSH service. We're going to turn it on and we're going to try to encrypt the VM

that is called Walter White. So, we're going to try to the Walter White VM here. And [clears throat] when we actually get to the ransomware, we're going to do something a little bit more sophisticated than the ESXi attack. So, we talked about how there are tools on ESXi like Netcad is is available by default. There are some other tools that are pretty nice for ransomware. And so we're going to go ahead and cd over that Walter White VM. We're going to do this big long command. I'll sort of explain it out for you. So first we're going to Grep. So we're going to list out all the files in this folder and that's going to give us all the

files we want to encrypt. Then we're going to use OpenSSL. OpenSSL is installed on ESXi by default. And so we're going to use OpenSS SLL to do our encryption. And we'll create encrypted copies of the files. And once those encrypted copies are finished, then we will use RM to delete those files. Now, you might notice that this requires us passing a password on the command line. And what the attacker could do here is just kill the logs after they do this. So, you can delete the logs on ESXi with a simple RM command. The logs are gone. So, there's not a lot of record of that password ever being passed on the command line or you could hide it in an

environment variable or something like that to make it obuscated a little bit. So, we'll go ahead and run our fileless living off the land ransomware attack. And we will see that all these files now have this little nuclear sign extension. That's because they've been encrypted just like the previous ESX example. If we try to spin up this virtual machine, we'll experience a very similar error just like last time. Operation failed. We're not able to run the VM. Again, a very bad day for the system administrator. Now, there's another really good example of lateral compromise that happened in 2024. This was a CVE that got assigned to ESXi systems where if you have your ESXi systems joined to Active Directory,

there's a special group called ESXi admins. And if you're a member of that group, you automatically get administrative privileges to all the ESXi host. and you can automatically log in to all the ESXi host with your AD credentials. So, it's a very bad separation of privileges. Typically, you don't need to have access to all the ESXi systems if you have access to AD. Now, obviously, if your AD is compromised, it's a very bad situation for an organization, but uh I would say that it's probably not a good thing that you can just turn on that access without stealing a password, stealing a credential to get onto the ESXi host themselves. So we'll see an example of

this. We'll go ahead and access the Windows host and we're going to already have access to the domain controller. So we're going to go to the active directory. We're going to create this ESXi admins group which had not been created before. We'll make it. We'll add our user to it. Our user is called Billy Jean. So we'll add Billy Jean to the group. And we'll go ahead and apply that. Say okay. And now what we'll do is log into ESXi. And so we'll log into ESXi using those same credentials for that AD account. Very straightforward, very easy. And so from here, you can start modifying the passwords for the default users on the ESXi system. So root [snorts] is

obviously a default user cuz it's sort of like Linux. CSXi is not really Linux, but it's sort of like Linux. And we'll go ahead and go to the users. We'll change the root users password, modify that password, and this will allow us to then log in to the ESXi host just like we did in some of those previous examples over SSH. And again, you can script all this. You can do it with a tool called Power Cly that we'll talk about shortly. Power Cly allows you to just run automated scripts against all these hosts, change passwords, can even create new user accounts if you want to. And now the attacker is going to run the

ransomware and they'll run uh very similarly that open SSL command that allows them to encrypt all the [snorts] files. So bad day once again. So lateral compromise is really tricky because again we're using capabilities that should be accessible to the administrators on these systems, but we're using them for bad. So, how do you detect that? How do you respond to that? Well, you can do patching and network segmentation. All that's important, but I'd say this is where you want some sort of runtime protection. Some sort of runtime protection that can detect when someone is on your ESXi box and they're doing something funky like downloading ransomware or running ransomware. I also think MFA is really important here. So

you want some sort of multiffactor authentication available so that if an attacker does get onto the box, they don't have the ability to just use a single password to SSH in or to log in over the UI. And so there are a couple different options you can do here with VMware environments. Can't really do a whole lot over SSH, although there are some tools that can add SSH to the ESXi host. One thing that you can also do is use a jump host. So you can use a jump host, have MFA on that, allow yourself to do MFA and then log into the ESXi host from that jump host. And then uh from the virtual center perspective, you

can use tools like Octo Duo to do MFA through that environment and that helps you a lot if your credentials get compromised, which is very often the first step for attackers when they go after these environments. I would also say don't join your ESXi systems to active directory. It's just a bad idea. It allows the attackers to get privileges on those systems that they normally shouldn't have, even if they do have access to AD. So, the next attack that we're going to talk about is the minor breach that happened in in May of last year. And again, this was not a ransomware attack. So, we're going to do some different stuff here. We're going to try to gain

persistence on an ESXi system. And so, how do you gain persistence on ESXi? It's a little tricky. Because ESXi does not have a file system that's normal like other operating systems. You don't have files that stick around. They are ephemeral. So if you reboot ESXi systems, many of the files disappear. There are certain locations where you can hide files. Those locations aren't typically used to execute things. So let's say you wanted to create like an init script or something like that. You can't put it in the place where you need your nit script to be to actually have it run on boot. What attackers can do is use something called a VIB. So VIB is a vSphere

installation bundle. It allows you to install software on ESXi systems and that does allow you to get persistence on the file system. So if the host is rebooted, you're good to go. And to do this, you have to tamper with something called the acceptance level. So VIBs have various different levels of acceptance. And you can think of them from an attacker's perspective. All you really care about is is the VIB signed or is it not signed? So has it been digitally signed by Broadcom or not? But there are all these different levels. Community supported, partner supported, VMware accepted, VMware certified. All those really define [snorts] different levels of support that are attached to that third party software.

So it's how Broadcom is going to support you if you have some sort of issue with that software. But again, from an attacker's perspective, all you really care about is whether this is signed or not. And the default value is partner supported. So the default value on all ESXi hosts is partner supported. But that requires us to have our VIB signed. So that's a challenge. Broadcom is not just going to sign our malware for us. We would have to do something very, very tricky or a key or something like that. So that's a little complicated. Maybe there's another way we can get our bib onto these hosts. Now, [snorts] another reason you're going to want to install a

VIB is because of a protection called Exec installed only. So, Exec installed only is enabled by default on newer versions of ESXi. Essentially, what it does is it says very much what it sounds like, I am only going to allow applications and executables to run on ESXi that are installed through a vSphere installation bundle. Anything else I'm just going to block. You're not going to be able to run it. So this would prevent you from running ransomware binaries, but it doesn't prevent you from running Python or busy box or any of those shell scripts that we were showing earlier. So living off the land is still a valid technique that you can use even if exec installed only

is present on the ESXi host. >> [snorts] >> So, another thing to keep aware of is if you have one of these unsigned bibs and you haven't done the appropriate things to the ESXi host, uh, if if secure boot is turned on, you'll get this nasty purple screen of death if you ever try to reboot it. And so, secure boot is not enabled that often. It should be enabled on most systems. There's really no reason not to, but a lot of people, I'd say about 75% of the VMware environments don't have secure secure boot turned on. But if they do, then you get this purple screen. If you try to install an unsigned vid and then you try to do a

reboot. So this is bad for persistence from the attacker's perspective. You want to be able to survive reboots. So how can we do this? So now what we'll try to do is we'll try to run some ransomware with exec installed only with secure boot with all those protections turned on. So, we're going after a hardened ESXi environment here, and we're going to try to run our ransomware. And we'll see on the right side of the screen, there's a log. Jump back to the end of the video there. And the log shows us Whoops. [snorts] Log shows us a message. That's very interesting. So, the log says execution of non-installed file prevented. So, this is the VM kernel

log. This is exec installed only in action. We tried to run our malware executable. Exec installed only prevented that malware from running. We're not able to execute our ransomware on the host. So to to know how to sort of bypass this sort of protection, you definitely have to understand what a vibly is. And so a vib is really just a targz file that has some XML metadata that defines how the files need to be laid out. It's got a very specific schema that you have to follow. But essentially, if you follow it properly, you can create your own VIBs. But of course, you can't sign them. They're not signed by Broadcom. And that has that issue with secure boot

and with Exec installed only. And so now what we're going to try to do is take a custom VIB that's not signed. We're going to try to install it on this this hardened ESXi host and we're going to try to run our malware through this unsigned VIB. And so we'll just do a naive software vib install command. and [snorts] we get issues. It says, you know, our bib has has problems. So, we could force it. We could just do a -f, which is a force parameter that bypasses some additional checks. And if we try to do that, get another issue about the acceptance level. The acceptance level of the bib is community because it is unsigned. So,

what are we going to do about this? This is definitely a challenge for us. Uh, we don't have the ability to sign our vib. So maybe we can just change the acceptance level on the host. So we'll try to change the acceptance level to community supported. And now we get an error. It says that secure boot is enabled. And this is kind of challenging for the attacker because it's hard to mess with the BIOS, especially if you're remote, right? If you're over in Russia or UK or wherever and you're attacking someone that's remote, you're not really able to get physical access to the system and change the BIOS settings. So what the attacker has to do now is they have to find a way

to get around these native protections, try to install this bib some other way. And to do this, you really just need to read the code. So ESX CLY is the command line parameter that's used to do a lot of administration on ESXi systems. And it's all written in Python. So you can pull that Python code off of ESXi. You can read it, try to understand it. And if you do, you will see that there are a couple checks. There's a get host acceptance function. There's a set host acceptance function. And under the hood, they run this function. That's run command advanced configuration. A few other parameters. And if you keep on digging, you keep on digging, what

you'll find is that that command uses something called the config store command line interface. So the config store is a runtime memory that's in a JSON structure and allows to to define the settings of the system. So we can see oops go ahead and jump ahead here. All right. So we have our config store CLI here and we're going to echo the community supported level into that JSON. Then we're going to load that JSON using the config store CL. So again, this is the runtime setting for the host. We're going to modify the runtime setting for the host and we're going to set it to be community supported. So when we do this now, if we get the acceptance level, we

see it's reported back to us as community supported. If we try to install the vib, this time it works. Even though we've hardened the system, even though we have secure boot enabled, exec installed only is enabled, we're able to install our vib. Now we're able to run our ransomware. It's going to encrypt all the files. The attacker is happy and the system administrator for the virtual machines is not very happy. So what can you do against this? Again, patch network segmentation, all that stuff is great, but this is another area where having some sort of runtime protection on the ESXi host is very useful. And so there are different options available for runtime protection. One of them is to use SIS

log. So sis log will allow you to gather the logs off the ESXi host. You can forward them to seams. You can analyze those seams to see behavior that could be associated with malware. The challenge with using SIS log is first there's a lot of data in those logs. Those logs contain all this data about just generic VMs spinning up, shutting down, generic information about the ESXi host working. Not a lot of it is security relevant. So you have to be able to parse out those security relevant details. And there's also the risk that if you're dealing with an attacker that somewhat knows what they're doing, it's very easy to tamper with CIS log to mess with the logs

themselves. And then your logs are just they're not that useful anymore because they've been tampered with because they've been modified. You're not able to get the information that you want out of the logs. Okay. So there a couple other types of attacks that we'll talk through. One of them is going to be a VM escape. So this is a concept where the attacker doesn't have access to the hypervisor. So all these other attacks we've been talking about, the attacker got access to the vSphere and the VMware control plane. Now the attacker only has access to the virtual machine itself. So you can imagine someone's running a VM, maybe it's a server, the attacker gets onto

the server and they generally have to have administrative privileges in that VM as well. And so there were a couple great examples of these sorts of attacks earlier this year. So in March of this year, there were a couple vulnerabilities that came out that were chained together to do a VM escape. And then recently in July, it was a pone to own where a bunch of CVEs came out. And in pone to own, it took VMware about 2 months to patch the vulnerabilities. And customer of ours had asked why why does it take so long to patch these issues when they're so severe. It's because Broadcom and VMware have partnerships with organizations like HPE and with

Dell. they have to do validation testing with all those different partners before they're actually able to release a patch. And so this is a big problem. If a new CVE comes out, if it's a zero day, if this escape to host exploit comes out, there's a delay that occurs because of this testing timeline. And then there's the delay of organizations just having to apply the patch, too. Sometimes these patches require a reboot, sometimes they don't. But if they do require a reboot, that's typically going to take a long time for an organization to deploy it unless they're pretty sophisticated. They have that whole process worked out well. So, traditionally, the way these exploits work is they're memory

corruption exploits. They'll often take advantage of device drivers or virtual devices and virtual interfaces to the ESXi host. This is an example from 2018 where there's a very good write up for it online. And what the attacker does is they write to a virtual network adapter. So that virtual network adapter is going to virtualize networking protocols and networking stacks and you can just write in some bogus addresses and tell it to write to those bogus addresses. So it's an arbitrary write vulnerability. Very straightforward. You just tell it to write to some arbitrary addresses. This allows you to then leak data later on and to launch further attacks to launch further memory corruption into the host.

And so the challenge here for detection is that you know you have EDR in your VM. You have a lot of protections. You get harden your VM. You have a lot of detection available in the VM. But these attacks are pretty esoteric. It's not like someone is running a netcat application and you okay I'm just going to detect netcat. No, they're doing something that's very sophisticated. it's very specific to some sort of device driver. Being able to do good detections for that that are reliable and robust uh across many different variations or iterations of the malware is very challenging. So, it can be very tough for a defensive organization to actually do detections on this type of

exploit because it's so sophisticated, so specialized on a specific area. It's not really generic in any way. Okay. So, now I'm going to talk about a couple other areas of research that are really strong. So one of them is about powerly abuse. Powerly is an API on ESXi systems that allows you to remotely administer them and remotely change things like user account information. You can add new user accounts. You can change user account passwords and you can start and stop services. So you can start and stop the SSH service for example. So typically when these attacks are run against enterprises, the attacker has access to powerly. They'll run Power Clyde through VSCenter and they'll just create a user across all

the different ESXi hosts. They'll change the password for that user to one that they know across all the ESXi hosts. They'll turn on SSH and they launch sort of a a scripted attack over SSH to run ransomware on the ESXi system itself. So, Andre Olsson does a really good talk on this. We're not going to watch this video because it's like an hour long. and he talks about ransomware and power cy abuse and he goes through a lot of details and a lot of background about the historical background of of how the attacks work and he's a he's a threat intelligence person that works at a company called trusk in in the Netherlands. He's very very

knowledgeable about VMware malware. Another really great area of research is on rogue VMs. So rogue VMs were something that were used in the MITER attack. The idea with a rogue VM is I am going to run a virtual machine from the command line on the ESXi host, but you're not going to be able to see it from the guey. So if you go into VSCenter or you go into the ESXi guey, you can't see that this VM is running. So this is like shadow IT nightmare. And what the attacker can do with a ghost VM is a they can get persistence in the environment. So they can run their command and control, they can run their

malware through that ghost VM and they can sit there for a long time. Another thing that they can do which is particularly nasty is they can read other VMDK files for other VMs. And so you can imagine someone spins up a rogue VM, it's a Windows system and they attach it to the VMDK for the domain controller. And once they have the domain controller and access to it, they try to leak passwords and password hashes to do further lateral movement within the network. And so there's a really great video by Christian Moan who's done a lot of research on rogue VMs and how those tactics and techniques work. But generally speaking, what you have to do is you have to set up a VMX

file which defines the details for the virtual machine. How the virtual machine runs, its name, all that sort of information. And then you just do this VMX application. You run the VMX application with that VMX file. And VMX is what runs virtual machines on ESXi. Okay. And this is some research by Valley Cyber's threat intelligence team. It was presented the Defcon Cloud Village this year. This is all about more detailed persistence methods. And so, as I talked about, there are certain files on the system that you could potentially modify to gain persistence, but there are tricks to it because the file system is not persistent. If you reboot the host, you're not actually able to keep those files around. So

there are different files you can modify or different files you can play with uh that and different techniques that you can use to keep those files present so that you can actually gain persistence on the ESXi host. So how do you defend yourself from these attacks? Uh you definitely want to patch you want to use virtual patching if you have the proper security tools available. Network segmentation is huge. I think runtime protection is a really important thing to have on your ESXi host because it allows you to detect and respond to some of those more detailed OpenSSL commands or some of those living off the land attacks that we were discussing. MFA is really important as well so that you can

prevent a single stolen credential from being the keys to the kingdom for your VMware environment. Monitoring SIS log, you can do it. It's certainly better than nothing. The challenge with it is what we discussed where the attacker can just tamper with those SIS log logs and there's a lot of logging that comes out. A lot of it is not security relevant. And then don't join ESXi the active directory. There are different guides that say you should do this. I say from a security perspective you definitely should not do this. You should keep that control plane and the VMware control plane as isolated and segregated as you possibly can. So last thing I would recommend to you is if you are working

at an organization, you should go on Showdown and you should see if you have any publicly accessible VMware assets. And so Shdan is free to use. You have to set up an account. But if you do set up an account, you can search by your organization. So let's say you work for Walmart, you can look for all of Walmart's publicly accessible VMware assets. And I just think this is a really great first step that anyone can take to make sure their VMware environment at least has this basic level of security around it. And hopefully you don't see something like this. So if you do some deeper research on Showdown like our threat intelligence team was doing a few weeks ago, you will

see ESXi hosts that are publicly accessible that also have been actually encrypted. So if you go to the web page for these hosts, you'll get ransom notes about pay us this much Bitcoin to get your VMware environment back online. And you'll also notice on this note in the bottom right, SSH is turned on and the firewall is turned off. So we talked about how SSH is often used in these attacks. We also talked about how firewalls are not enough for defending in these issues. So it's definitely the case that you want to make sure that you're doing more than just firewall prevention on your VMware environments. So any questions that I can answer? >> Uh question regarding the rogue VMs. So

do you know what SIS log messages would be generated when they're uh executing that command? So because we're piping everything to our SIM, so if I create a detection, that would be great. >> Yes. So there are certain commands that that do get exported over SIS log. In fact, there's something uh about the DCUI process that gets run when a VM gets run. So I can get that to you offline. If you want to send me an email at this email address, I can give you some some pointers. >> Okay. In the case of of either the RC being a listening service or a guest escape exploit though, would you expect to find anything in the command logs or

anything in the log at all? >> Yeah. So, probably not because those sorts of activities are just they're just not going to be logged because you're doing something that's outside the scope of traditional running a VM or traditional application usage on the ESXi host. So, escape to host exploits are one of the scariest. They're definitely sophisticated. Hackers are not going to use those very often. and you're typically going to be like a big bank or a big DoD organization that's going to get hit with that sort of thing. But if you do and if you are one of those organizations, there's not a lot that you can do to trace it. >> Um, for home VMs or like just like club

like a cyber club, how would you suggest um how would you suggest protecting those? still the same suggestions outside of like not doing permanently ad or is that >> yeah I think that the same sorts of approaches definitely apply for for home networks as a as opposed to enterprise networks. Are you using ESXi or are you using some other virtualization technology? >> Personally I'm using ESXi or sorry personally I'm using something else but um our cyber club does use ESXi. >> Okay cool. Well, I would say that for personal usage, if you're not using ESXi, a lot of these systems that are hypervisors, tier type one hypervisors, except for um the Windows variant, which the name is escaping me right now, but

uh a lot of them are Linux based. So, if you just follow generic Linux hardening guides, you can do very well just protecting your your home lab that way. But when it comes to your cyber team's lab, I would definitely recommend following these different recommendations that I provided in the talk. And then there are hardening guides for ESXi available online too. Those can be very effective and very helpful as well. >> Thank you. >> When you what do you mean by runtime security? You [clears throat] made some kind of tooling for ESX. >> Yes. So we built the first EDR that's capable running on ESXi host. That's what Valley Cyber has done. It's particularly innovative in the in the

last year. Uh so you talked about exec installed only uh flags and uh specifically showcase how they can be bypassed by dynamically changing the acceptance level that honestly sounds like a bug in VMware and uh I was wondering if they have done something to counteract that or like patch each other back like that or is it is their stance on it like uh nobody should be able to get into the ESXi console to actually get to the point where they can bypass these. >> Yeah, that's a great point. So that that attack we did where we changed the acceptance level in the host that seems like a CVE. It seems like a big issue with with the VMware environment. And I

will say that that VMware definitely is focused on that harden your systems approach and prevent access into the control plane. And so you're kind of like, hey, if someone gets shell access on the ESXi system, you add a lot because the other thing to recognize is on Linux, you'll have different user accounts with different privilege levels. On ESXi, you have different user accounts, but they all run as root as root as UID0. So they all have super user privileges. And the other thing to recognize is that there's some research that was put out by Anders, one of the researchers that I talked about where he was able to bypass some protections that allowed the um the attacker to create uh

malicious accounts on the system that they had access to that they could use passwords to log in. And what Broadcom did in response is they didn't issue a CVE. He feels like they should have issued a CDE, but what they did is on their documentation, they just said this can be bypassed. They just put that on the documentation page. So I I think that uh might be sort of the same thing with a config store client because it is you're just using the the native tools on the system as they're intended to be used. They might not issue a CVE for this sort of thing. >> So you mentioned SIS log monitoring is one of the main ways to like this but I

think as you know if you worked on incidents once they've touched ESXi it's usually like within seconds they're already encrypted by microvisor. So by the time you get those logs off the machine and you've triggered detection um you're already kind of cooked. So I was wondering if your um your EDR here actually has preventive capabilities on ESXi. >> Yeah. Yeah. Exactly. So >> So yeah, I I um our EDR does have preventive capabilities. We can definitely block things from happening on the ESXi host and we have anti-tampering capabilities that prevent the attacker from modifying with it or changing the security. And you know, SIS log is definitely a good option if you don't want to go that way or you have

some other uh approach to securing their systems. But the challenge with it is what you just said. These attacks are fast and the logs can be tampered with very easily. Okay folks, well enjoy the rest of the conference. Enjoy the afterparty and it's been a great bides. [applause]