PG - Messing with Forensic Analysts: Modifying VSS Snapshots - James Clawson

all right thank you all right hi everybody my name is James and we're gonna be talking about some pretty cool stuff today before I actually get started though just a quick question so by show of hands how many of you guys have any experience in forensics oh wow a lot of people okay that's awesome so about a year ago I took a friendís class and when we got to the section on VSS I remember just being amazed at how much awesome stuff you can pull out of those snapshots there is just is so much information that people don't even realize is being recorded and honestly when I saw that I thought to myself man you're guys this job is just way too

easy so I decided to take a few weeks of my time to make your guys's jobs a little bit harder no I am I am kidding a little bit you know I do love you guys I know a few forensics people but snapshots are surprisingly easy to mess with and we're gonna see that in just a second all right so where we actually be talking about today first we uncover the basics of what VSS actually is and what it's used for we're gonna cover a few reasons why you should care about it if you don't already then we're gonna proceed on to what it actually looks like on the disk itself this is where we start to

get into some of the really cool technical details after that we're gonna go about on to describe how we actually modify a snapshot though this is the really cool stuff then we'll go ahead and actually demo that and we'll go ahead and modify a snapshot and finally we'll talk about a few different ways you can tell if a snapshot that you're looking at has actually been modified or not so first of all what is VSS according to Microsoft VSS stands for volume Shadow Copy Service I'm not sure how that acronym works out this is what they say if you have an issue you take it up with them so the two areas that most users will

encounter VSS is previous versions and System Restore those two features make use of VSS to actually work so previous versions if you accidentally delete a file or you know modify it or something you can pull an old version out of a snapshot system restoring if you know a system update fails and you installed bad driver you can just roll back the whole system it's pretty cool so what VSS actually does is it keeps track of changes that are made to the files to enable this so add a disk level what's actually happening is before you actually write data onto a disk the VSS service actually copies the old data before it's overwritten and stores it in the

snapshot space this is called copy-on-write this appears in a lot of places not just VSS and so let's actually go ahead and take a look at what that looks like here we have our hard drive and this in blue space is the space that's reserved for snapshots the space may or may not have anything in it right now we don't know honestly we don't care too much so when we want to write some data to the hard drive we figure out where we want to write it in this case the little highlighted block before we write it we copy our data over into the snapshot space then we proceed with our write we write our new data

onto the hard drive where we decided to and at this point we have our new data safely on the disk and our old data is safely in the snapshot so this is what Windows is doing constantly in the background as you're using the OS as you're writing to the disk so how does Windows actually take this and go back to the snapshot though let's go ahead and take a look at this so we start with our disk how we just finished in our last example and what we do is we create an overlay of sorts from the snapshot so that grey block ends up being placed in the overlay where it used to be which is

where the green block is and when we applied this overlay on top of the current disk we end up with a view of the disk when the snapshot itself was actually taken

all right so back to more basics about VSS it's enabled by default on every system since Vista it creates snapshots automatically daily on Vista that was used in a blog to space they changed it to weekly on Windows 7 and later the system will also take snapshots automatically when you install system updates new drivers that sort of thing so it can roll back if needed and it deletes old snapshots as necessary it only uses a certain amount of disk space for those so an interesting little side note is that this is not just used for System Restore in previous versions for example MS sequel server actually uses VSS to make backups while it's running so the sequel server will briefly put

the database into a consistent state it'll pause writes and it tells the VSS service to take a snapshot and as soon as the service is done taking a snapshot the sequel server goes back to writing to the database doing all of its normal stuff and then sequel server can take its time writing out that snapshot of data to the hard drive it can compress it send it wherever do whatever it wants so this is actually one really interesting case the two groups of people who know the most about VSS are forensics people and DBAs I'm really not kidding like if you go and talk to your DBA about VSS they will go on and on for

20 or 30 minutes I made that mistake one time I'm not gonna make it again another little interesting side note is actually that early ransomware authors didn't know about VSS just like most users so it turns out with the early ransomware you could actually just right-click on your encrypted file select previous versions and then restore an unencrypted copy of the file your yeah so then ransomware authors obviously caught on to that pretty quickly and they delete snapshots now but for a little while there you go why should you actually care about it I mean if you're not forensics person or a DBA well first of all it's enabled by default so it's pretty much going to be

running on any Windows system you actually interact with one of the most important things about this though is a secure deletion of files becomes really difficult or much more difficult at least because when you intentionally right over file to try destroy its contents the VSS service is actually going ahead and making a copy of the dady you're trying to overwrite right before you overwrite it so you go out of your way to destroy data and windows is going out of its way to preserve that data in a snapshot and once data is in a snapshot you cannot individually remove or modify files until now at least the best you can do is delete a snapshot

over at your disk space and hope that covers it and because most users don't know about VSS and don't realize all the stuff it's in there snapshots frequently contain evidence of crimes or simply private information tax documents any sort of other personal stuff let's go ahead and actually look at a few cases where VSS came into play our first case is new versus the state of Georgia I'm sorry to start the first example off with a pedophile but it is what it is so this guy had a whole bunch of child porn on the system he knew the cops were coming for him he ended up deleting all of the child porn and wrote over all

this free disk space ordinarily without VSS this probably would have destroyed the data in this case the police take his computer the first thing they do is look at the snapshots and they pull out all the child porn he just tried to destroy he obviously got convicted the next case we have is USA versus Michael Thomas this is a really interesting case this was actually in the news a few years ago so this guy he deleted a whole bunch of his employers data right before he quit his job and so at this point you're probably thinking you know that cops went to his workstation or his desktop and look through his snapshots and found evidence of his guilt that's

not what happened in this case he actually goes into court and he employed what's referred to as a novel legal theory he went into court and he freely admitted that he destroyed all of that data he argued though that all the data he deleted was safely stored in snapshots so it didn't actually impact the availability or integrity of the data um as you would expect this didn't work he was convicted if you ever get arrested do not use that defense

it's not just used in criminal trials though the next case we have is actually an example of a civil case HMS Holdings Corporation vs. aren't at all this is sort of similar to the last case but in this case the former employees were accused of stealing confidential information rather than destroying it so the defendants allowed for forensic exam examiner's to look at their system the forensics diameters look through snapshots of course they don't find any of the files that the defendants are accused of stealing they do however find evidence of file scratch file shredding tools and they even find logs that indicate those tools were used to destroy the files they were accused of stealing so even if the data itself

isn't in the snapshot you might still be out luck okay let's get on to the good technical stuff what does it actually look like on the disk first of all Microsoft will not tell us they have published absolutely no documentation about the on disk format they published plenty of documentation for users for admins for developers so if you want know how to use it great if you want to know how it works not so great fortunately there's this dude named Joaquim Metz and he will tell us he spent months of his own time researching the on disk format as well as researching some more cool technical details about it and the really cool part about this is he didn't just keep

this information to itself he actually published detailed documentation online so this is how we know via set how VSS works it's basically because of him not only did he published occupation he also created an open source library called Lib B Shadow the links at the bottom of the page is also on a later slide and this open source library actually allows us to read from snapshots it's a cross-platform library pretty cool so basically he spent months so I could spend weeks so you can spend hours to frame your next-door neighbor and I'm obviously just kidding it only takes minutes not hours okay so that's great and all but really what does it look like on the disk it's pretty much a

bunch of linked lists and 16 kilobyte blocks someone at Microsoft just absolutely fell in love with 16 kilobyte blocks if you don't know how big some it's probably 16 kilobytes so the on disk format isn't really all that complex unfortunately though we don't know everything about it when we encounter something we don't fully understand we kind of just wing it and it generally works and kind of just go with it as long as it's working don't ask too many questions all right it consists of an NTFS header a catalog and stores each store holds a single snapshot one snapshot one store one store one snapshot they're basically synonyms the catalog contains information about the stores

slash snapshots and the NTFS header simply tells us where to find the catalog it's going to take a look at that on the disk so we here we have the NTFS header at the beginning of the drive this isn't a known fixed location we have the catalog and in this example we have two snapshots store one and soar to the NTFS header tells us where to find the catalog the catalog tells us where to find the snapshots so the whole point of this thing is for us to actually modify snapshots so once we've located the snapshots we don't really care about the NTFS header or the catalog anymore they hold some cool information but we're not concerned

about them we're really just concerned about the snapshots themselves the next obvious question is what is in a store it's a whole bunch of stuff you've got a header you've got data block list block range list there's a ton of stuff we don't need to be concerned about most of it though we really only need to worry about the data block list and the data blocks themselves the data block list it's a linked list structure and it keeps track of changes that are made to the snapshot the data blocks are just the data themselves it's just 16 kilobytes of raw data that was saved in the snapshot the data block list elements each element of that data

block list has a bunch of block descriptors the block descriptors are what actually described the changes so let's go ahead and take a look at this so it makes a bit more sense so in this case we have an example snapshot with three data block list elements each one simply points to the next in the list and each one holds a whole bunch of block descriptors it's 508 but that's not really important and each block descriptor describes a single 16 kilobyte change all right so let's one more time let's go over how Windows saves data to a snapshot but in a little bit more detail this time we want to write some data to our hard

drive first we copy the old data to the snapshot we write our new data now this old data that we wrote to the snapshot it's referred to as a data block just 16 kilobytes of raw data now what we do we create a new block descriptor and this new block descriptor simply tells us where the old data used to be it just tells us that where the gray block is now it used to be located where the green block is

so when we create our snapshot overlay that little arrow showing where the gray block used to be a single block descriptor describes that arrow it tells us how to draw the arrow basically and of course it gets more complicated there are a whole bunch of different types of block descriptors fortunately they all serve the same basic purpose though I would like to point out though that the unknown type of block descriptor in that list this is an example where we don't fully understand everything we kind of just do our best and it usually works okay so let's get on to the really good stuff we've seen how Windows reads and writes to snapshots now let's see

how we actually tamper with an old snapshot so it's actually pretty simple in this case we start with a good base so we start with lid V shadow slap on a few thousand extra lines of code spend hours fixing really stupid bugs the next thing you know you have right support it's a little more complex than that but that's the gist of it okay so what we're really doing though is we're basically doing a windows does but we're doing it backwards when we want to write data to a snapshot instead of writing it to the disk and copying the old data we just write our data directly into the snapshot itself then we create a block

descriptor that says that the data in the snapshot used to be somewhere else we're gonna go ahead and take a look at this so this is how we tamper with a snapshot we have our disk as it currently exists at the top we have our snapshot overlay when you applied the overlay you get the disk when the snapshot was taken so we work backwards and we want our snapshot to look like this we want to add those three evil red blocks into our snapshot in order to do this we work backwards we have to have those three red blocks appear in our snapshot overlay at the correct location in order to do that those three red

blocks need to be in the snapshot space and we create block descriptors to draw the slayer arrows so we know to put those three red blocks into the overlay in the correct location and just like before we apply our overlay and now snapshot looks how we want not how it actually was so as I mentioned it's not quite as simple as just adding on code and spending hours doing stupid stuff there were a lot of issues that came across I'm not gonna spend six hours covering all this stuff you guys don't have the time I don't care to all the explanations and examples though are on github the link will be on later on the

slides okay so let's cover the actual demo let's see it so in this case I'm going to be demonstrating the Linux version of Bob Jesus demo failure with the video I did not sacrifice to the demo gods let's take a look at this I'm gonna be demonstrating the Linux version actually the Windows versions in progress we just have an issue it's drivers and writing two of them raw disks so let's go and take a look so this is our windows vm and we have drive Y this is the disk we're gonna be messing with so the guy who uses this machine he loves sysinternals he's always talking about non-stop it's really annoying I love sysinternals too but seriously

shut up about it we're gonna delete sysinternals from his disk because I know he's just gonna go back into a snapshot to restore an old version so let's go ahead and take a look at our previous versions this is our snapshot data and if we go ahead and look at the oldest snapshot we can open that up we'll see sysinternals is still there now you could just right-click on the directory and actually select a restore and it would actually push it back to the current volume so we can see in this case all the sysinternals data is there so we're gonna go ahead and shut down the vm and we're actually going to tamper with it the snapshot off the vm

so in just one second we are going to be using our modified version of V shadow mount there's the - W option was added that enables write support in this case the snapshots from that disc are going to appear in MN t / VSS work if we go ahead and look in there we see VSS 1 VSS - those are our two snapshots so we can actually mount these files as if they were discs themselves and it's basically a view of the snapshot so the file the snapshot we want modify is a VSS one but it's gonna go ahead and mount that using normal tools it's going to be read/write and let's go ahead and take a look at it

so if we see sysinternals is there we can go ahead and take a look inside the directory all the files are still here this is just what we saw in Windows but we're gonna start modifying it we're gonna get rid of these files and I for one love Bodeen this guy needs to love bugging - let's copy a boat picture after we do that we can see that boat pictures there we go ahead and unmount our snapshot we unmount our VSS working directory from V shadow mount and then let's go ahead and start up the VM again

let's go ahead and skip through a minute or two of system and starting up and we're gonna go ahead and open up that snapshot and take a look at what's inside so please go ahead and right-click properties previous versions and we're gonna open up the same snapshot we just looked at we open it up we take a look inside the sysinternals directory those files are gone our boat picture is now there he does need to buy a boat obviously instead of a boat picture we could have put you know other funny cat pictures I just chose the boat picture I'm going to talk about that in a second that's a really good point so what you just

mentioned how can we tell the snapshots been modified so in this example I just used normal tools to copy a file in there so when I copied that file in there if you would look closely you would have noticed that the snapshot was from a few days ago the file I just copied has timestamps from a few minutes ago when I just copied the file this is one way that you can tell this is one of the easy ways look for anomalous timestamps files that have timestamps that don't belong they should raise a red flag other things that you can look for you can look for event logs you can look for timestamps and other files any

known activity that the user performed if you happen to know for sure that the computer was turned off at a certain time and there's a file that has a timestamp from that time that might be a red flag for you so you look through your normal stuff and you just look for stuff that's out of place there are also plenty of hard ways to find out though and what it really comes down to is Windows does some funky things when trying to snapshots livvie shadow does some funky things when it's reading from snapshots and my modifications to Libby Shadow does some funky things when it's writing to snapshots and those funky things don't always line up

so this is definitely a lot harder if you are curious you can look through the code you can look through documentation on it and there will definitely be more details so links the good stuff so on the top we have this is a me on github so the code and the slide show are going to be up there it's not there right the second will be later today and then below that we have the regular unmodified Lib be shadow this is just straight from Joaquin Metz his documentation is also in that repo if you you know if you want to find out more about VSS on the disc look through the source code and his documentation

I'll go ahead and eat this up for a second so it's gonna depend on the exact tools you use if you just use regular tools it's going to update the timestamps oh oh you're talking up modifying the timestamps yes if you modify the timestamp so they fit in it's not necessarily gonna be obvious not an easy way you can look for the harder ways yeah yes yeah NTFS has like 12 timestamps alright just a really quick thanks walkie Metz for doing all the hard work and documenting it sliding just swoop in talk about take the credit Chad Tilburg he's the guy who actually talked the forensics class I mentioned earlier Patrick Thomas my b-sides mentor

if you understood the content of this presentation it's his fault alright are there any questions and also source code email address behind you given limited understanding of the CSS format beyond what you've said my I my understanding is it's possible to essentially reconstruct timelines from it and it's a fairly standard forensic process using essentially the set of entries to identify a spirit a turn of changes have you looked at the problem of creating anomalous changes such that it's very obvious exactly what you changed because essentially blips in existence and then modifies out it just doesn't fit the time flow and we thought about how to fix that problem yeah that's definitely an issue I think you might be talking

about two different about two different things there one is basically if you modify a single times single snapshot what about the snapshots on either side of it yes that is something you'd have to handle manually the code doesn't really do anything about that at the moment so you modify a single snapshot it's not going to be on either side yeah yep you can delete not really yeah what other data emitted is preserved when you take a snapshot does it include extended attribute set does it include alternate data streams okay so that's a good question so all the data they can snapshot it it actually gets snapshotted on the block level so whenever you edit a file whatever actual changes occur to

the disc on the block level get backed up so it doesn't VSS itself doesn't actually care about file systems or anything like that anything that gets changed on the disk will get saved in the snapshot attributes alternate data streams all of that and that's all the time we have thank you very much all right to our speaker [Applause]