"Don’t Lookaside or you’ll miss it: Turning a Hyper-V cache miss into 200k cash" Leo Adrian, Cbr23

uh we have our talk now this rated really well in the review process and I'm really looking forward to it as well uh don't look aside or you'll miss it turning a hyperv cash Miss into 200k of cash Leo Adrien let's welcome him to the [Applause] stage cool hi everyone thanks for coming uh so hi I'm Leo um I used to be a pent tester but for the last year or so I've been independent researcher focusing on Windows um and a shout out to the link team cuz I started there less than less than 5 years ago and now I'm standing here finding Odes so yeah great team over there um cool so this talk is a little bit of a journey um definitely a bit more winding and uh and precarious than I thought it would be from the start it follows the tale of some cool research I did back at the very end of last year and through January um and as I mentioned I'm an independent security researcher so my goal was I wanted to find a really cool or really good hyperv buug um earn some Bounty money and that would set me up for for the rest of the year to to do some other research that was maybe less profitable um but like all good research it took a bit more of a securest path than I was expecting and so hopefully uh by sharing that with you today um I can remind everyone you don't have to be an ultra Elite hacker who started when you were 5 years old um you can also just be someone who's very persistent Not Afraid not afraid to try something new and and make lots of mistakes along the way um and I saw this tweet the other day like 3 days ago I think and I just had to had to put it in here because this is exactly what it's like um yeah that you have this amazing goal in mind and and nothing goes quite the way you expect um so hyperv what is it um hyperv is Microsoft's virtualization solution so it's what they use to run VMS so both in terms of the Azure Cloud if you spin up a VM there that's running on top of hyperv but also if you've just got a Windows 10 Windows 11 laptop you want to run some VMS hyperv comes uh by default you do have to in it's it's free you'd have to install it um but you don't need to pay anything extra it's just kind of there alongside the operating system um and it also is used for some security features we're not going to talk about that but like secure isolation mode for Elsa stuff like that um but what does the virtualization platform actually look like um there's a lot of components it's pretty complicated you definitely don't have to uh remember all of this but there's sort of three main components uh to hyperv there's what they call a root partition um this is just your host operating system so technically when you turn on Windows 11 with hyperv installed that's actually like a VM that you're seeing with your desktop it's just a very special one they call the root partition partition's the windows term for VM um I'll probably just use them interchangeably but they mean the same thing um then you've got child partitions this is just this is just VMS like if you spin up a Linux VM on windows with hyperv that's a child partition um and then underneath them all is this the thing called the hypervisor and that kind of ties them together it does does some really cool lowlevel magic that's not actually what we're going to be talking about attacking today we're looking at the root partition that's kind of like the overall thing um and I think legally if you talk about hyperv everyone has to include this diagram because because every talk I've seen so about hyperb has it um so thanks Microsoft um so now we have some idea what it is what are the security properties why would we care about it as attackers um well hyperv the the hypervisor and and sort of the root partition that manages things um they broker all the hard the interaction between VMS um and the and the hardware so if you spin up a VM and you're like I want to give that thing two CPUs and 4 gig of RAM but my laptop's got 12 CPUs and 16 gig of RAM well hyperv has got mechanisms in there to make sure that the guest VM only sees the things you want it to see in terms of hardware and also if you've got several VMS running like you would in a cloud it also Brokers the the devices and the hardware so that the VMS can't interact with each other when they shouldn't they can't see any other any other systems running on there um but as we just saw it's a pretty large and complex looking thing right so we're thinking about security we're also thinking there's probably there's probably going to be some bugs in there and Microsoft knows all this too right so they offer a pretty nice amount of money uh if you're able to find something that even looks like a guest to host Escape so it's $250,000 if you actually have one with an exploit even if you don't write the exploit part of it even if you've just got a bug that looks like it could be exploitable they'll pay up to $200,000 um and even before I started this research there were other researchers who' publicly said yeah they gave me the money so that's kind of why I chose this this bounty program to to put some time into um so I'm not going to cover all the components here but I sort of broke it down into like what the attacks surfaces of hyperv the kind of functionality you can interact with from from the guest um we're going to really focus on that root partition kernel so that's the kernel driv is running in the the host OS that you're using to manage all the VMS um but there's a lot of attack surface in in the userland so in sort of the actual processes not in the kernel stuff and then the hypervisor itself is also a really cool and complex uh attack surface but not something I've di dived into directly um so most specifically I want to about talk about this thing called vmus um vmus is a what they call a power virtualization framework um so normally when you have a virtual machine right it's it's an operating system you boot it up and it goes oh cool I'm running right I'm alive um I want to talk to some Hardware like what's my CPU how much RAM have I got are there devices available to me like hard drives graphics cards like network network cards things like that and so back in the day what they would do is they would emulate these devices right you would write some software that pretended to be a hard drive like a spinning platter disc drive um really really complex to do really finicky if you get it wrong the operating system is going who the the hard drive stopped working like it's freaking out um and it's also really inefficient uh so over time they develop this this better idea called par virtualization where you install some special drivers into into the VM and onto the host that's running it um and these drivers are kind of aware that they're being virtualized right so you turn on the VM and it goes oh hey I'm running inside hyperv so instead of just acting like I'm talking to a real hard disk I'm going to use this special set of drivers specifically for hyperv uh they're tune to be way more efficient um and to be able to talk to the host uh in a way where we we sort of both sides know that we're not really using Hardware we're kind of in this in this software abstraction um and vmus supports a few different device types we're going to we're going to look at storage mostly today um it does require Hardware support so if you have an incredibly old CPU that doesn't have support for the virtualization extensions you can't do this stuff but we're talking like more than a decade old at this point um so the fundamental technology that that vmus is built on um and that makes it really efficient compared to to some of the emulation techniques is the idea of shared memory so if you've ever used like shared memory mappings in in Windows or Linux programming the idea is essentially that you've got your your physical memory so like your RAM um and we do some cool stuff with with virtual memory and Page table mappings uh to map the same bit of Ram uh into sort of the root partition into the host's memory and into the guest VM um so the addresses like the Little Numbers you the hex numbers you'll see would be completely different uh but actually when the CPU does its all its tricks to resolve the page tables they're pointing to the same place so this means one side can just write write to a piece of memory and the other side sees it pretty much instantaneously so it's a really really efficient way of communicating and that's kind of the high level idea of how how vmus uh becomes really efficient is um any communication of of sort of mass data is just done by setting up one of these these shared buffers and what will happen is the guest will like say hey I've got this buffer at physical address two3 and it'll send it through the hypervisor to the host and the host will go to the hypervisor and say hey the guest has sent me two3 that's not a real address can you tell me what the real address is and the hypervisor will do some magic and go sure the real address is actually 456 and so the host can map that address and now they're looking at the same thing um and yeah really efficient um and if you want to think about it in kind of like an OSI model way um yeah so vmbs kind of replaces like your physical layer and on top of that each device has its own specific protocol so who've got Vore nvsp they're really small we don't really worry too much about those and on top of that you have your more generic protocols that you might be familiar with like SCSI RN is like a Microsoft specific networking over USB protocol that they just sort of crowbar in there um but then on top of that is tcpip and you have your applications above that so it's it's kind of how you can think about it and um the the lower level uh protocols incapsulate the higher ones right so as you as you're passing these you unwrap the lower layers and pass them off the stack um and before I go any further I I I want to give a shout out to this guy Peter havav um because his blog post here is kind of what gave me the impetus for research we're going to talk about today um I had previously looked at some hyperv stuff but this was kind of what got me back into it and going hey like this is something I want to look at again um so this describes some very successful research he' done he found like 10 12 bugs over a couple of years in hyperv um and he goes through his methodology here um and essentially at a high level his idea was like he's going to do fuzzing but instead of just doing data fuzzing like hey I'm sending a message from A to B and I'm going to mutate the message he said it was to to fuzz the state of the hypervisor so it's not that the individual messages being messed with but like sending messages in a weird order or across multiple threads trying to find race conditions and um like this kind of stateful fuzzing something that's been done before with things like szalla but I'd never seen it talked about in the context of hyperv it was really successful for him and so I was kind of like this is cool like I I have some maybe ideas of of where I could take this and and and do this kind of stuff myself um cool yeah so now to get into something a bit more specific um so so I after reading this blog I sort of had like a bit of inspiration a bit of a plan I was like okay I'm gonna replicate peters's work on on this sort of stateful fuzzing um but I'm going to focus on the virtual storage layer because it wasn't something i' looked at before I was like I've got this great I'm going to implement a a multi-threaded generative fuzzer written in Rust like I didn't have ai back then but I probably crowbar some AI in now right like really cutting edge stuff it's going to be it's going to be awesome um and so I spent some time reverse engineering this storage stack trying to like okay figure out what are the different components what are the actual drivers I'm working with I learned a lot about SCSI it turns out like that's just your generic storage protocol but um yeah there's actually like some SCSI commands that are in there but they're not documented unless you're like the member of like a secretive Industry Group it's like I guess it's like the secret to fast copying is not something they can give away for free um and I wrote like a little harness uh using the Linux kernel so the nice thing about hyperv is all the the sort of guest VM stuff for the Linux kernel is completely open source U Microsoft have upstreamed into the Linux kernel because obviously they want Linux to work really well in Azure um and so I was able to go in there and play with that Cod a bit and use it to start sending messages directly through this vmus thing into the the virtual storage layer that I kind of wanted to fuzz and Target it's like yeah cool things are going well this this seems like a great idea and then I get into the fuzzer I'm like fuzzing is awesome like fuzzing is really cool right now it's in Rust as well I love rust so I spent quite a while like writing this this generative fuzzer and what I mean by generative is again we're not mutating some some particular like data or packet instead we're generating packets that should be valid but we're sending them in maybe a weird order or we're sending them across a couple of different threads at the same time and hoping to trigger a race condition um and and this was going to happen like in in the Linux uh in a Linux VM and we're sending it down to the down to the the hyperv uh host and the konel um and so this is basically me at this point I'm like yeah cool like great idea I'm definitely going to find like a thousand bugs with this thing um it's crazy fast like I'm doing a lot of benchmarking on it but I'm not really doing the fuzzing while I develop it you know I'm like having to recompile the Linux kernel constantly is really slow so what I do is I sort of write some stubs to like yeah I can definitely send messages from the colonel to the host but I'm going to do most of this stuff just by like writing some Rust and have it output to standard out or write to a file like you know that's that's kind of what I'm doing when I'm developing this thing and and it's crazy fast right like I'm getting great results it's looking awesome um so I want to just have a quick interlude and refer to uh to Mark out's 2022 offensive con keynote where he's talking about the helpfulness or unhelpfulness of writing tooling as part of the vul research process so I'm going to quote him I've seen a lot of times in my career people start doing vulnerability research and they're like I'm just going to automate this thing and then they spend forever doing that and they forget why they were even doing it and they make this huge complex program that might not even be particularly useful in real life because they're things they think elegant and are going to work in their mind don't actually work in practice um yeah so I've spent probably a good month writing this this cool fuzzer in Rust and it's so fast uh and I eventually like pluged the pieces together like cool I connected up to my little Linux kernel harness to like start sending packets down to the host get some crashes um get a crash but for those of you who haven't seen this particular kind of message before this is the message you get when you make the Linux kernel really upset uh and so basically um it was it was pretty uh immediately obvious that uh actually I hadn't thought about a bunch of stuff and all I was doing was crashing my own guest konel um I had this amazing idea of like yeah this is this is going to work really well it's going to be so so fast but fundamentally i' I'd failed to understand some of the some of the protocols and Technologies in use um in my sort of keenness to start this project uh and my attempt to just crowbar a fuzzer into this Linux kernel stuff wasn't working um yeah so I I've mentioned do is create a buggy guest VM it sounds like you could probably still get a cve for it these days but um not what I'm trying to do uh and in all honesty I felt a bit silly right like I'd spent a bit of time I I got really tunnel visioned into this idea of writing this particular fuzzer but I didn't give up and I started to like figure out okay what's going wrong and it turned out that um you know there's this VM buus layer that I briefly touched on that's kind of the transport mechanism and I hadn't really thought about that all I was like I'm going to generate all this data and just send it and I was like okay but like how is it sent and is there like a cue and there's some ring buffers and there's actually like you know there's there's a threshold Beyond which it can't send more data and you have to wait for it to catch up um and essentially I was just overflowing all of these things on the Linux side and then the kernel couldn't do any storage like any disc iio and it was getting upset and it was crashing so like I slowly started to figure out like oh okay actually just like trying to make things as fast as possible in this case is is not going to work um but a couple of days into this mess I'm just you know constantly testing the fuzz like maybe it'll magically work this time um and actually I suddenly get a crash right this is a real one um this is a wind debug a screenshot of wind debug so this is a crash I got in the actual host Kel I'm like wow okay we're back on like we have got a real guest to host crash um but where this came out of nowhere like what I thought my fuzzer wasn't working um so this point I drop everything else around the fuzzer like yeah I'm not working on that thing anymore um and try to triage this crash and figure it out um so this is a stack Trace sorry it's probably a little bit small but the the first thing that jumps out me so there's the crashing instruction everything above that is like the windows crash Handler we don't care about that at all um but looking at that that instruction and the function and the the driver that's involved and what's below it um is there actually no storage stuff there at all so this doesn't involve any part of the host storage deack it's not in the like SCSI passing or anything instead it's in this VM bus layer that I had completely you know ignored and and not bothered to understand um and so I don't really have any any background for like or context for like what's going on here um so I spend you know luckily in like the first sort of 30 minutes I'm able to get a like a very pinpoint understanding of like oh okay this is a use after free bug there's there's like a link list being iterated and one of the one of the nodes in the list has been freed and I sort of get a very rough idea of what's going on but I don't really have any context for like what this vmb km CLR driver is or what it's doing um so I have a little bit information about this crash but really if I want to report it to Microsoft and and certainly get the full Bounty I need to be able to reproduce it um unfortunately as previously discussed my highly Advanced rust based fuzzer does not help um it's basically non-deterministic because of the the really buggy stuff it's doing on the Linux side so I run out a few more times it just crashes the guest colel it's not helpful like I've basically fluked my way into this into this cool guest to host crash but if I want to reproduce it I'm actually going to have to understand what's going on uh and somehow like manually trigger the conditions for this thing to happen again um so yeah definitely not a deterministic fuzzer which is is a problem um but one thing I find that's really helpful is to start to document assumptions with this stuff like okay well how does this actually work what's going on and so starting at a really simple level like the guest is sending some data some stuff's happening and then the host is crashing right and I can even from what I mentioned before I've done a little bit of triage immediately I can sort