GF - Tomb Raider - Automating Data Recovery and Digital Forensics

good afternoon everyone thank you for attending uh this afternoon's presentation uh blue will be presenting today on tomb raider automating data recovery and digital forensics so i'm the absolute honor in uh going ahead and starting this presentation of course what presentation would be complete without a few prior announcements such as going ahead and thanking you for being here we do like to take a moment thank our sponsors whom with this would be absolutely impossible to be here with you and present in particular we would like to thank our diamond sponsors uh which are lastpass and palo alto networks uh we do have a couple of gold sponsors too that we would like to go ahead and say thank you to and that includes amazon intel and google today it's their support along with other sponsors donors and volunteers that make this event possible so a quick reminder please do keep your masks up during the presentation uh we are trying to keep everyone safe so we appreciate your support uh cell phones uh go ahead and please make sure your cell phones are not only uh turned into like a vibrate mode but also please try and keep them in your pockets if possible we are recording this going ahead so we'll get we'll get it posted online uh there's really no need for you to go ahead and take photos or videos you'll have the ability to see all this afterwards uh and just a reminder of blue uh b-sides uh not blues sorry uh b-sides uh policies we do request that you do not take photographs without explicit permission of everybody in the photo of course we are trying to be sensitive to those who are in unique situations that prefer not to be photographed so please try not to take photos uh here without explicit permission of everybody in the photograph so uh also as a reminder um well i was ahead of myself i got all of that all right so without further ado i will go ahead and turn it over to blue blue thank you so much for being here and uh have a nice afternoon thank you very much glad to be here man what did you do i knew i forgot something okay so um please if you go if you're going to go ahead and ask a question there is a microphone in the middle of the room it's not that we can't hear you it's because it's being recorded those who watch the recording afterwards will not be able to hear the question so we do request if you're going to ask a question uh towards the i guess the end of the you know q a session please come up and use the microphone in the middle of the room so we can capture that of course blue it's always nice if you can repeat the question when they say it just so that we go ahead and capture it as well so thank you yeah i'll repeat it no problem this is still good for mike right you guys can hear me back of the room you guys can hear me too okay great uh cool without further further ado's i'm really glad to be here and uh thank you guys all for coming so i'm gonna be talking about a tool i made today called tomb raider much like the game there will be a few jokes about that but overall uh i hope the naming of it becomes obvious soon enough uh automating data recovery and a little bit of digital forensics as well so first obligatory introduction my name is blue yes i like the color if i were green i would die this guy he gets it his difficult life pronouns are she her whatever but uh yeah and that's the problem you get the bad kermit we don't want to see that otherwise we're good to go so thank you all i like hard drives and data recovery that's the topic of today i started on my own broken hard drives i think many of us have these and uh we just had it laying around and i figured i'll figure out how to do this and work from there since then i've started doing all manner of other ones the uh ones i could find at local hacker spaces in the dump in the trash anything really and uh yeah let's talk about how you do what i do so first we're going to talk about an overview of everything that this tool does we're going to talk about the steps of data recovery and a little bit of the background of that and then walk you through how we're going to automate those tools those steps and improve them with tomb raider so we start with hardware prep you have a broken hard drive how do you go from that to an interface you can work with afterwards how do you image the drive we'll talk about what that means and how to do it then how do you possibly recover a file system if you've lost it and if that doesn't work or if it does work how can you also get extra deleted files determine what a random piece of memory is something known as file carbon which i'll go into in a minute and then afterwards some stuff where to write tomb raider really shines is uh post-processing so handling all of this data that you get from a given hard drive you might have terabytes of data from even a small like 500 gigabyte hard drive how do we deal with that how do we sort it out and actually find out what's ours what's not or if you're dealing with a random dumpster drive hard drive how do you figure out what's interesting and what's not how do you figure out what's actual user data and not just microsoft system files additionally if you're looking for it you can also crypto salvage and look for if there's any evidence of cryptocurrency on that hardware so first again why would blue do this well there's lots and lots of reasons you are already here so i i kind of succeeded congratulations you've now you're now trapped you can now have to learn about data recovery uh just kidding you can leave if you want but whatever point is i usually don't like asking why because i mean i mean i'm already here i'm already trying to learn it i digress a lot of us don't want to pay tons of money just when we could do it ourselves we have a lot of broken hard drives i did i didn't want to have to shell out 500 bucks for them to maybe not even get anything or especially if i have my own private data on there i don't want to just give it to someone and be like hey try and get this the only way to know if you've actually gotten it is to get your data so that is a privacy concern there's lots of reasons you might also just want to explore like i've done where you find random hard drives that otherwise would have been disposed entirely and figure out what was on them what is the story there additionally if you are interested in digital forensics i am happy to announce that you cannot do forensics if you don't have a hard drive with data on it if you can't get the data off of it how are you going to do forensics on it you can determine oh this hard drive got hit with a sledgehammer and that's about it but if you can't recover the actual data from it it's a bit of a bummer so i consider this a stepping stone additionally a better programmer and a better hacker so first what you can do here's what you can do with data recovery more than 90 of the problems with hard drives with minimal effort you can just solve them really easily from my experience with ones that are literally e-waste thrown away and i have no reason to believe that most of them were i have no reason to believe that they were function functional or rather i should say many of the hard drives i found i found little errors on them that were really easily fixed but i presume people threw them away because they didn't know how to fix them and now with this tool i was just automating it and going through it and finding all this stuff that people didn't realize was still there they thought it was just oh it's not booting this is happening why is that there are of course the other 10 though 5 which are harder software problems or hardware itself like if you did hit it with a sledgehammer you might have this problem where you damaged the board on top of it or in case of even worse cases you really destroyed it in which case well i can't cover that in this talk the other five percent as well unfortunately i only have so much time so ask me about it later or in the q a if you like that other other five percent the five percent that we cannot really recover we've actually seen some death gun talks about them if it's properly encrypted if it's completely overridden with zeros and or random data or just new files it's gonna be gone if it's blown up and the platters are broken if it's melted into slag well we don't really yet have the ability to do that unless maybe you have a really serious microscope like in crazy resources and patients additionally some stuff with solid state drives i will cover that later if you guys want to ask questions and talk about it um anyways intentionally destroyed drives this was a hard drive that i found in the san francisco uh dump and um you'll notice a lot of things about this first off all the pins are really weirdly bent and the ribbon cables on the left are cut the there's cables that are directly cut without much damage to the rest of it which is a little unusual you would expect if this just got dropped on the stairs it wouldn't really look like this and that's because it i have reason to believe this was intentionally destroyed um and they well they intentionally tried to destroy it because in reality all i had to do was find another hard drive with the same model replace the board and then i could read off of it just fine because the underlying thing the underlying data storage device that had all the files was still there it was a little beaten up they presumably threw it against the wall or something but it was working just fine and on that interesting things this hard drive contained data from the san francisco water this guy who's on the san francisco water transit authority we had some information on boat requirements and specifics for vessel voyages and all manner of things that had to be ready for additionally random stuff like this presentation which has wingdings in the title i'm not entirely sure and uh making fairies a viable transit option this guy's article on feet of fairies which is a little bit of a weird name but i'm not going to judge so steve here feels very strongly about fairies he would choose berries if one day the day came down to it so good for him um other stuff you might find interesting account info first name last name home addresses phone numbers this is a massive address book it keeps going and these are just random names of somebody who presumably was an accountant or secretary or just a home just a normal person and they left this there thinking it would be gone it was not gone there are a scary amount of data on these things if you're interested in finding stuff like that i don't encourage using it for bad but if you're interested and you just like finding stuff like this like i do give it a shot uh oh yeah psychiatric records of this guy's medical history that was pretty rough that was quite a story i blurted out but yeah that's a thing on a brighter note i also found copies of the room by tommy wiseau and other wonderful movies and films like that where you can just get for free because why not it's on someone's hard drive if you've already got stuff on your hard drive and someone else got it congratulations now they can get it using this tool you also find glitched images like this which can be uh which can have little stories of their own like i like the soccer guys who are like really upset about their image getting glitched out and they're trying to argue it back i have no idea what happened with the aol image on the left or the ducks or whatever that is curious things like this interesting and foreboding ones like this and the terrifying monstrosity on the top left that looks into your soul uh little darla has a treat for you is what that one in the middle says which is let's continue to something less scary oh god no anyways this guy i have no idea where he came from but he was also on a hard drive his name is craig and at first i was terrified of craig because i was like i mean no offense craig but look at him um but he actually just has a little spatula and a little skillet on the left there and i realized he's just making a little breaky and he's misunderstood and now i love him anyways you can also find dogecoin at bitcoin whatever um slightly less valuable now sorry guys but i have found a few of those and made money but i'm not sure if that or craig made me happier all this from these intentionally destroyed drops you may have one like this you may have one that's less damaged so with that said let's get back into how to do this how tomb raider can automate it how you can find your own little treasures how you can recover your own personal data stuff that is important to you we'll give a brief description of each bid and how it works because i sure didn't know all these steps beforehand and then we'll cover how tomb raider automates it we'll cover how we can make that faster optimize things speed things up and uh walk through it to do that i'm going to employ what i call the library analogy and to explain that first imagine you have a big library this library is your storage device your hard drive it has all of your documents all your downloads all your pictures somewhere in it you have it organized in your own specific way using a catalog an index just like you might go to a library and look up where is the fiction or non-fiction where are the uh books by jules verne or uh edgar allan poes poetry you have documents downloads desktop pictures media music everything like that somewhere in your library of your hard drive this is your file system and without it it becomes uh really really hard to find stuff there's a lot of books in there so this is how we're going to represent it and with that said talk about deleting files no data cut recovery talk would be complete without deleted files right so library storage device books the data and our catalog is our file system what happens when you actually delete a file in your file system the normal way throw in the recycle bin empty the recycle bin throw in the trash into the crash whatever whatever yada yada well unfortunately for those who really want it gone that is not going to actually delete anything that is going to remove it from the file system but not off of the hard drive itself it's never deleted it's only overwritten this is as if you remove something from your catalog from the system that stores it and lets you find it but not from the shelf like that that never happens unless you really really try hard to get it to do that sometimes it's easier but by default that's not the behavior and default behavior is uh what most people end up using most people who i've read these drives from are using specifically and if it's still there well let's just you know take the bookshelf and handle that later so let's get your device ready for tomb raider to run on i usually just plug it in to my desktop i recommend not plugging in random drives into your work desktop because that is basically just as bad as plugging in random flash drives into your desktop and we know that's a bad let's not do that i usually just use a normal motherboard connection on my desktop to do this you can use a lot of adapters however that work just fine on most computers and you may not be able to recognize it in your file system itself but once you find the identifier for it which i described on the github page and at the end of this presentation you can see the link to that page where you can download the software and deploy it in everything that's all documented along with the software it's got usage instructions and even if your file system doesn't recognize it that's how you can find it it works best on linux i did develop it on linux so yes sorry maybe future development in the future it's so much easier in linux i can just download all this stuff and there's only a handful of commands to set it up if i have the time i'd love to deploy it for other os's but uh you can also just help me make code requests on the github repository please anyways imaging once we've got it plugged in once we've got it recognized using tomb raider and instructions on the repository we want to image the drive what is that well basically we take the entire library why would we do that because it's pretty big and we don't really want to just have a big disk that we have to plug in all the time just to read it it might get damaged it might get like it might get damaged it might actually get worse over time the more we mess with it and that's no good at all we wanted to produce the same results every time we want to be able to copy it if we can so we don't lose anything else so tomb raider uses a tool called safe copy to incrementally get as much data as possible from the drive it uses this incrementally makes a bunch of passes through it tries to get as much as possible even from areas that get damaged or corrupted safeguard is a wonderful tool so thanks to that it's the only part that requires the physical drive after this you can unplug it throw it in a blender maybe don't but you know you can anyways now that we've got the entire library in a file on our computer we want to try and get the catalog for it file system recovery we want to find our way to recognize where our documents are to find out if we still have pictures what are the audio files it's pretty helpful to have our catalog we've likely spent a lot of time organizing it it's not explicitly necessary because the books are all still there we can still take the bookshelves but it's very very helpful and we want to find it if we can so we start with that it yeah metadata we use testdisk to get a set of these files yes this is another lovely tool we make use of a lot of them to get all of the files that it can from a variety of supported file systems and uh recover those if at all possible because that of course if we don't have to go through those bookshelves on our own those bookshelves on our own let's not because we'll see later that we'll see in just a second on that topic it's not too fun so start with test disk afterwards we do file carbon this is if we did have to go through it manually this is if we don't have a file system and we did take the bookshelves if we do that we want to look through them for stuff that still looks like books we want to discard the catalog entirely and say screw that i don't trust anything i'm just going to look through all of my bookshelves even if it looks empty and see what i can find and that's what we call file carbon you are carving out unallocated space you're carving out of this empty space this randomized data maybe actual meaningful data we use headers footers there's a lot of different mechanisms to possibly get this data from it it's super fun let's go through an example just to show how fun it is so jpeg this is uh some of the markers for jpeg from wikipedia i don't know if you can see this but it's okay if you can't there's just a lot of uh different indicators of uh hexadecimal encodings whatever stuff that represents the start of the image at the end of the image and here's an example image in this example image you can see all sorts of different jumbles of hexadecimal characters and uh if anyone can spot the start of the image in this on the right here let me know and uh raise your hand i'd love to see it as an exercise yo what's up crap i i can't hear you okay wow they found it it's in the top left good eye um there's this random octet right here this random set that is the beginning of it obviously it starts with that and then there's another code the way later down in the list that goes right next to that and that's another indicator that this is a jpeg very helpful not very easy to spot we do have to go through all this list checking for these things and then at the end of the image we have to look around for the code that indicates it's over because we not only have to know if we're carving something out where it starts but