Saturday Keynote: Scanlime (Micah Scott) (@scanlime)

I I guess I just wanted to talk reverse engineering like you know like like the introduction said I I think reverse engineering is is pretty important and I like spending a lot of time doing it and I think maybe it would be useful to have a slightly different way of thinking about it you know something that's less a very specialty tool that we use for kind of crafting these like really awesome exploits or whatever and more just part of learning how computers work so yeah that's pretty much for going but I mean like hawkers are so cool right like like I don't know anybody else feel like a hacker here I mean I don't know like these guys are

cool yeah I mean like these guys are cool right I think I think a lot of them are actually jerks and I don't know maybe a lot of us are jerks too but I think the main thing is that we're kind of misunderstood and there are a lot of different things that people mean by hacking now and I I think you know this this sort of like media view of like you know we're all really cool and breaking into stuff and we can just like do anything we want with computers in five minutes it's not really that helpful but I grew up with stuff like this so it's still pretty cool these are also cool hackers

I don't know where this slide came from though Rick's a hacker too like I said a lot of hackers are jerks but you know he can like build a portal gun in five minutes so people think he's really cool I just think it's like this part is so not real though like you don't the reason someone is kind of powerful in this computer world is not because they can do things quickly it's usually because they have the patience to be doing them at all so I find I like I love working Morty I think these characters are great but they're not necessarily how I want us to be thinking about this going forward this is like

more the stuff I grew up with like when I think about hacker I think of like people who don't look photogenic on TV people who like right Linux kernel drivers and stuff and like that was what I was doing as a kid and I thought that was really cool I don't know a lot of these people are also jerks but we try to fix that but I felt really at home here like I would find other people who did the same thing we would hang out on IRC and that was that was like how I learned computers you know I I didn't have a whole lot of documentation and there wasn't necessarily a lot of documentation that

existed for a lot of the stuff I was interested in like I would find a game that I really liked and like how does this even render and I would have no idea and maybe years later I would find oh someone else reverse engineered this and wrote a book on it but at that point like just taking apart that game probably would have been the only way to figure out how it actually worked so yeah these I mean these hackers I think the core value they have that I really fell in love with was this idea of just doing a huge amount of work and then just putting it out there for free but specifically in a form that other people

could build on and I just really like that a lot and I still find myself putting a lot of emphasis into that but yeah it's I don't know there are a lot of things different things people mean by hacker and different things people mean by hacks and like I I think a lot of the stuff we do is useful for understanding how computers work enough to figure out where the flaws are and what we've built and that's that's great but I also think a lot of hacks are not as useful for that and then I think that some of the stuff that I'm the most interested in is maybe more useful for understanding computers than for kind of

breaking them open so I don't know that's that's kind of the complexity here like I find that a lot of the stuff that I am very interested in is super applicable to security but not just about security and so people at security conferences are interested but I think maybe everybody should be doing this kind of stuff so I don't know so this is me I I spent a lot of time just tinkering with stuff this is kind of an artist's conception of what I was like pretty much everyday at a previous job and you know so as a kid I got started in this because I just had a lot of time to just dump into programming stuff and

my dad had this 8086 computer that he would bring home to do like mathematical modeling stuff at home and I would just like you know get some time to play with that computer and he would give me a book about it and I would start with with that kind of little bit of knowledge but it's so different than today with the Internet where you can just instantly get the answer to anything you want to search for back then it seemed like knowledge was so scarce that like focusing on the ways that you can gather knowledge about anything seemed super important and I was also just drawn to areas where the manufacturers or other people who are

building stuff just kind of had some incentives not to tell you how things worked and I think it was also just really fun I thought it was really fun to explore new worlds and you know every piece of technology that you can see inside of its you know it's kind of like a little game world it's like a little you know MMORPG or something you can get inside of and and just explore for a huge amount of time so it's it's a lot like a game and so actually I wasn't really that interested in games as a kid it turned out it was a lot more like you know my brother would be playing the

games and then I would get really curious how they would work but this I don't know if anybody's familiar with the game genie there's this thing you would stick in between the Nintendo cartridge and the system and so actually this is when you start to learn that oh the Nintendo system is a computer and the cartridge is a program and this is a thing that edit to the program in real time and then you can start to you can start to kind of get some idea of what's actually going on inside this black box and kind of replacing pieces of it in order to to get a sense for what effect that had and just it's a lot like

science I think doing experiments trying to get some sense of what's going on inside this black box just by just by making small you know small experiments or small tests that they try to cut across one particular aspect of it so you know I I had some other incentive to reverse engineer videogames I got really interested in the controller ports for some reason because that was just a nice interface that was exposed that you could just kind of dig into and figure out how to deal with so I had a lot of fun with this is a an adapter that goes from GameCube to n64 because I really didn't like playing Super Smash Brothers with the awful joystick on the n64

and this is before the Gamecube Super Smash came out and yeah it was it was really fun and I think this is especially the kind of project that really takes a lot of people by surprise because they think oh it's just some buttons like how could that be hard and if you if you were doing this you know maybe some years earlier and this was an Atari joystick every button would have its own wire and it would be super easy but this was just an interesting project partly because the n64 controller was so weird I don't know if anybody remembers that controller but it actually had expansion pack on the back of the controller and you could put like

savegame memory cards in there a lot of times people would end up just sticking a rumble pack in there which was like the add-on vibration motor for the controller and so what I wanted was to emulate the vibration feature but the vibration feature was actually this separate add-on so it was actually a matter of figuring out how that memory bus works and then being able to emulate that bus and so I mean this this kind of thing inspired me at a pretty early age and I just kept doing it as much as I could and and video games in particular just seemed like a really good way to stay interested in this stuff enough to

put enough time into it you can see on the right that was kind of my set up for a lot of this you know old school oscilloscope and that little breadboard had a couple of pic microcontrollers on it I think one of them was actually the the work-in-progress and then the other one was a debug tool that would take the packets and dump them out over serial because I didn't have a logic analyzer at the time and that's that's one of those things that may be worth mentioning is that some of this stuff does take tools like it could be a lot easier if you have just the right piece of equipment and that can seem like a

barrier to entry but that's been that's never really been that high of a barrier if you can kind of cleverly work around those requirements you know if you can make your own tools in cases where maybe you you would have liked bought a logic analyzer for like a lot of money back then it would have been maybe a lot more cost effective if you had the time to just program a microcontroller it's a sample that one particular piece of data you care about so I think it was something where maybe my interest in reverse engineering was also coinciding with just more access and and I think at this point there's just like no excuse not to be able to take things apart

there are so many good tools out there and there's there's so many new ways of both sharing and finding information you know about how these things work there's another video game related project to had a lot of fun with I don't know if anybody knows this game robot Odyssey it was kind of a sequel to Rocky's boots which is another kind of classic like circuit building sort of game I get really into this as a kid we had it for Dass and I never actually finished it because it's a super hard game it's actually uh it's it's like an adventure game you're this you're this person who like gets stuck in this robo robo tropi

list like sewer that's run by you know robots that don't want you to escape and you find a couple of friendly robots but the only way you can really interact with them is by building circuits inside them and that's how you get through these puzzles so I I got really into taking this game apart and actually figured out a bunch of the file formats and reverse engineered a lot of the code that bottom left actually the game normally worked with xt style keyboards like on a modern computer you can't even really use the keys correctly because it's made for such an old keyboard but i tried adding mouse support because i thought that was fun and that was mostly

just oh we'll add some code to the game's main loop and now I know where the object tables are in RAM so we can just compare the mouse cursor and that was that was really fun I even took this a little bit further and decided to try porting it to the DS and this was this was actually really fun because this is actually running multiple copies of the game simultaneously so the top screen is kind of the canonical instance and it's got a graphics driver that takes the like CGA frame buffer and scales it out to the DS size and color converts and all that but on the bottom screen you can see these little icons representing

the state of the characters and so those are actually made by scraping the memory for those objects into another instance of the game which just renders that sprite and because it's all like designed to run on a four megahertz 8086 with a good enough binary translator you can just run so many copies of this in a TS this was another just fun example of something I did a long time ago that kind of set me on this path just wanting to take apart everything this was when I was living in like an actual house a while ago and we had like a circuit breaker box that you could install stuff in and and there was this product where you could

put some little current clamps in your breaker and then it would actually send information about your power usage over the powerline and you'd have a little box you plug in in your kitchen or whatever and tell you how many kilowatt hours you're using and I wanted to make a kilowatt clock that would tick once per kilowatt second and it was kind of a stupid idea but it was also an excuse to reverse engineer this protocol and it was I would find out that that box actually had just a very purpose-built she had been it that just is a powerline modem that just receives these sorry receives these packets but I thought it was a lot more fun to just build one

using op amps and stuff so you know this this sort of project taught me a lot and then I tried to document it as best I could to try to to try to help other people understand kind of similar aspects of Technology this is this sorry this is of course isn't one that I made a lot of you are probably familiar with this is a video yeah so this is called the visual 6502 I guess one thing I forgot to mention about that robot Odyssey project is that that was one of the very few times where I've really felt like I could just reverse engineer all of this it's like I I have you know

maybe 100 kilobytes of code here I'll just sit down and figure out what all this does but usually that's just not practical usually your whole system is just so complicated that you have to pick a little slice of it to kind of cut it along but this is one of my favorite examples of something that does go really deep they they actually just scanned the 6502 dye and converted all the layer images back into polygons that were accurate enough that you can actually simulate it in your browser and you know and actually see each individual metal wire changing state so I think I think projects like this are super inspiring because they show you just how deep you can cut through the

stack and still have an understanding of how things work if you you know if you choose the simple enough platform but most software I feel like is much more like Lego blocks you know I mean the real world is is this complicated mishmash of interfaces and none of them really worked the way they say they work and they're all leaky but we have to kind of ignore that when we build software usually and so you end up with all these pieces where you know what they do because you've used them a lot or you you've read the instructions but that often doesn't really tell you what's going on inside if it's open source maybe that's like you took a

bunch of little Lego blocks and built them and then like potted them in clear epoxy so you still usually use it as one big brick but you can look inside it if you want a lot of software isn't even like that though and you don't have any visibility into what's going on you just have the documentation you just have you know maybe source code for part of it and I mean software folks love love these blocks I mean none of the blocks are real these diagrams are full of lies they're always full of lies and they're only useful for understanding just the very exterior layers about technology you know it's like you open a datasheet

and this will tell you roughly what's in the design or you you look at an API and this tells you kind of how the architects intended it to work but this was always something that came out of somebody's head near the beginning of the design process and it's always full of full of problems whether they're intentional or not I like to call out software block diagrams because they're particularly bad because software is just so easy to turn into spaghetti and the hardware has some you know often has some physical constraints like if you're making a circuit board usually that's not going to just be crazy complex because that actually costs money every time you make one and there are

limitations about how much complexity you can pack into system-on-chip designs to some extent but those are really becoming much more like software where you can just have this explosion of complexity that there's not really a good way to deal with in the way you would deal with something by kind of wrapping your head completely around it you have to be able to find a way to kind of dig into one specific part of the thing in order to understand it when it gets this complex so yeah I mean this this Lego block world is super fun it's like a great place to hang out if you're Engineering something like if you're developing a product or if you're

prototyping then you want blocks that you can trust you don't want to have to take everything apart to figure out how it works and so that's like a case where you really wouldn't want to have to do any of this but I I often try to avoid the stuff that that has such an easy solution so I often find myself like needing to to figure out what what is underneath that floor that we've all been relying on being solid the floor is never solid you know we always we always kind of try to figure out well what can we sit on it feels comfortable enough to actually build on and there isn't you know you can just keep going down right

we still don't really know how physics work we probably won't understand physics for centuries more and but we still build these incredibly complicated systems by assuming we can start at some particular point and so some people start with semiconductor physics and some people start with you know very log and a lot of people just start with whatever is in their language runtime or you know instructions on their CPUs instruction set but you always in defining leaks and that a lot of times you can just like search Stack Overflow and somebody will have already figured out what's wrong with it but you you get into more esoteric areas and you just can't rely on that you actually have to

do some research and and kind of do science on this bug in order to figure out why it's actually happening and that's that's reverse engineering like when you have to start taking apart those blocks instead of just reading what's on the exterior so one thing that I did a lot of for kind of reverse engineering for interoperability I think I think anybody who's made software that other software sits on top of probably has had to do some kind of like very difficult kind of reverse engineering based debugging because if you have this complicated thing you don't understand like in this case portal and you have a bug in the video driver that portal is running on then you might just have a

crash in a bunch of code that you have no documentation for and I would I would end up spending a lot of time just trying to figure out well in this complicated system where I can only have visibility into small pieces of it how do you actually understand enough of it to debug the whole system and yeah VM with VMs you're just asking for this kind of complexity because it's sort of like you're taking that or that everyone else had been relying on and like jacking it up on stilts and and now it's kind of shaky and you're kind of just cheering up like anytime it's about to fall it's yeah it's usually not so great and and then the

floors the floors aren't so they're really bad sometimes like sometimes you think you can rely on something as simple as DRAM and then you realize that well actually they didn't plan for all the different ways you can access DRM and all the refresh timings and you end up with something like row hammer where and if you know about row hammer that's a way of by accessing DRAM in a particular pattern which will depend on your motherboard chipset and the DRAM itself you can cause it to you know you basically cause it to not refresh fast enough to keep up with the amount you're disturbing it and you can cause bits to flip and and I just cuts through all

these levels of abstraction we build up on top of that that stable floor and and that's not necessarily something that you can afford to be thinking of when you're in that like product development like building things out of like well-worn Lego blocks kind of land but if you if something's starting to go wrong or like maybe if you're doing something where a failure like this would be really catastrophic then it helps to actually every once in a while just take that stack of blocks that you've been sitting on top of and actually look down and try to take some of them apart to see if what you're doing is really working so I don't know

the metaphor that I've been thinking about reverse engineering lately it's kind of like a chisel it's this this kind of slow kind of messy tool that you know it doesn't really take a lot of a lot of training to just start splitting apart a log and I think that's a good analogy like I don't think reverse engineering takes a lot of special skill to start out with I think everyone should be doing this I think it mostly just requires kind of knowing enough about what you're doing to to want to invest the time in it and I think that's where a lot of people get stuck is that they see this thing which seems complicated and they don't really know

where to start and and so they just maybe think I I don't have time for this I've just got to search Stack Overflow which is you look the right answer but maybe maybe it's worth fine some of those projects where you can actually start to break things open and understand them on your own to to get more of that more of that experience and when you do split the system open along any of these you know anything's kind of fault lines you get all this access to whatever's on both sides of that interface so this is this is something I had a lot of fun with hey I mean so much of reverse engineering is even figuring

out well I've got this chisel which by that I mean just some way of opening up an interface and where do I even put it there's just so much to choose from on a system like this so this is the Nintendo DS Nintendo DSi mainboard actually I think twl is the codename of Twilight and there's just so much going on here like there's a bounty of test points right like some of those are bound to be interesting and I I wanted to get a sense of how this worked so that I could help the homebrew community run you know like run their own code on the Nintendo DS the video games were sort of an

interesting thing to get into because a lot of the information that you would need in order to write your own code for the game system is actually being intentionally obscured but for kind of silly reasons like Nintendo wants their licensing fees and they don't want you know random folks writing code for their platforms and I think that's kind of a silly reason so I'm happy to kind of work against that when I can and so in a sense I mean this is reverse engineering for interoperability because I really wanted to run my own stuff on this platform but it's also reverse engineering because the platform is kind of interesting and it's kind of a way to

to try reverse engineering something that has more of these features that are trying to keep you from getting in the previous DES actually was pretty simple to get into that was when Nintendo took their Game Boy Advance and just added a second processor in a second display it didn't really change much else but the DSi it took a lot more cues from how Nintendo built the Wii where the we actually has a chain of trust and it has a separate crypto processor and and they they ported some of that over to the DSi at least enough that there was a chain of trust from Buu and it was a lot less trivial to just run random code on it so you know the

test points were interesting we didn't find him that much that was interesting on those though I tried replicating somebody else's work where they found a vulnerability in the savegame files for this game healthy cooking coach it's called cook cook hack so a bunch of people are buying this cartridge it was it's not a good game it's really not it's very buggy which is good for us because that game has already been signed by Nintendo like we have this chain of trust and you can run unsigned games but then they only have the previous platforms level of functionalities to get half the RAM you don't get the SD card slot or the camera so we wanted to open up homebrew for the

entire system which meant bypassing this code signing system they had so one way of doing that was to find you know it was like some kind of buffer overflow in the savegame format and so you could trigger this just by clipping on to that SPI memory in the cartridge and writing your own data onto there exploiting the bug once but I was especially interested in coming up with a system for repeatedly being able to do this so I would started using an FPGA to emulate the RAM or the the flash memory chip and then I could even see where the where the platform is reading from and try changing those values really quickly and then just change it reboot change it

reboot so that was nice I didn't end up getting that much farther into it via the spi memory there though because even when you get full code execution in that cooking game the cooking game actually does not have a lot of privileges so we couldn't use the SD card slot for example and I think it was still running at like half the memory or something so then then you have to break out the magnet wire and despite this board having so many test points it turned out the signals that were most interesting did not have test points so that's the system on chip on the left that thing has two ARM cores it's basically the

Gameboy Advance and the second processor they added in the DS and the GPU so basically three cores and a memory arbiter and some caching there's some RAM and the SOC but most of the RAM is next door on an eight Meg SDRAM and then the non-volatile storage that Ian which is basically an SD card in a chip and I was actually one of the first things we looked at but that was encrypted and we didn't have an easy way to get the keys of that the way this actually ended up leading though was to this kind of increasingly weird FPGA rig where this was actually just taking all the signals that went between the RAM

and the system-on-chip and bringing them out to an FPGA and the first step of this was actually just just logging the data so I would slow down the clock on the DS as much as possible which was limited by the the clock multipliers on the chip so once you slow it down too much the clock just doesn't want to run but I slowed it down as much as I could and then I actually just barely got it slow enough where I could take that Ram data in real time so every time it would read or write to main memory not the cache not the local memory but the actual external memory then it would stream that back over USB so it was just

barely enough for USB to bandwidth and that was that was actually great I learned a lot that way but then I took it one step further and ended up needing a slightly faster FPGA and doing some circuit board surgery and that was actually to get in between one of the chip enable lines on the on the SDRAM and the processor so with that you could actually in the middle of a memory packet you could kind of switch off the DRAM and then answer that memory packet yourself and so that was actually how I implemented kind of patching arbitrary code into memory so you know again there's there's a bunch of internal memory there's like a ROM that it boots

off of there's internal SRAM which it tries to keep all the sensitive data in but at some point it is actually relying on data in the external Ram during boot so it was a matter of kind of changing some of those data structures I forgot whether it was a pointer or some code or there's something in there you can patch and then gain control over the code execution put in some of your own code and then you can say copy all the internal Rams external Ram I know you've got all the crypto keys so that was it was a great project I learned a lot from it this was really before I was in the

habit that I am now of trying to document the work that I'm doing though so yeah I have like this super super messy Dropbox folder with like 10 gigabytes of binaries from this and it's terrible and so that isn't something that a whole lot of other people have been able to build on but I really wish I would have done it more like that and I think I think you do see this in especially the video game reversing community that a lot of people reverse stuff and then just never release it and there's a lot of good reasons for that I mean some people just don't want to get the video game companies mad some people

don't help the pirates and those are great those are great concerns like I've personally just not release stuff because I don't want it to be used for piracy but I think it's nice to be able to find projects that you can document more thoroughly so I went a little deeper into that with graphics tablets kind of recently and I don't I don't entirely know why I chose graphics tablets I just think they're cool like them and I I think a lot of people use them but don't really know that much about how they work and haven't tried to just take them apart and figure out how to like how to really get in there and

modify it so this was one of the first things where I I did a kind of thorough series of reverse engineering projects on it and sure I didn't just do it all on video and that's that's been my preferred way of sharing this stuff these days is I take video I live stream everything I can but then I edit together videos that try to go through the process and you know sometimes I don't know sometimes reverse engineering will will lead down this impossibly deep rabbit hole where you could just spend so much time getting some knowledge that may or may not actually help oh yeah I forgot this head what you're boarding here yeah that's great yeah so this was

actually part of the video where I was kind of explaining it's kind of explaining the link between putting graphics tablets and RFID cards where they both use very similar technology where you know it's not really radio it's more like a transformer because you're dealing with these near-field couple systems but yeah this was this was a another project where I used just a regular AVR microcontroller to emulate an RFID card and so just kind of showing the connection between those I think a lot of this turns out to be about experiments though so a lot of what I was showing in these videos was was kind of doing science on this device so like you know for example here I've

got a coil on the left that's sending out these quick impulses so just like very fast rising magnetic pulses that'll if there's something resonant nearby so I explained these resonant circuits a bit like electric pendulums where you have in this case and it's an inductance and a capacitance that kind of keep juggling energy back and forth between magnetic and electrical and so in this case you can actually start to use this to see what the pen is doing and you can see oh the pen is resonating because it's actually taking some of the energy in that peak and storing it and then re releasing it as this oscillation and so even just using little experiments like

that you can just start to see what the pen is doing without even having to take it apart or destroy anything and then here I was noticing that if you if you touch the end of the pen and like press the pressure sensor that actually changes the resonant frequency and you can kind of get an idea of what it's doing so these this was one of the tablets I dug into it's made by who yun the really the really popular brand everyone knows about is welcome and so I was working on a Wacom tablet a little bit later but just to start out I just wanted to take the cheapest tablet I could find and tear it apart and and

just show people how it worked and it turns out these were actually a lot simpler they they just use this pen that's basically a little a little transmitter that just transmits at a single frequency and the frequency changes depending on the sensors and the buttons on the pen this is the level that people are more used to dealing with tablets out I think this is so tablets are USB devices you might have heard about USB HID or human interface device spec it's it's a it's like how the computer knows when you plug in a USB device the USB device uses the you know the data structures in this hid spec to describe how the physical world

maps to the bytes in the packet and so we can get some information about how it works just from looking at the USB protocol layer but to get an idea what the pen is really doing yeah the pen needs pen needs an oscilloscope really so so here I'm taking a closer look at the data that I'm seeing actually between the pen and the tablet tablet so in this case you're starting to see I don't know this is a Wacom tablet yeah so I started to get into the Wacom because you know they they're more complicated I think people had more questions about them because it's more just like a more common brand and so the

Kooyong tablet was a nice thing to just get into and it had that continuous sine wave but this was a Wacom Intuos Pro and so you can see it's actually a lot more complicated there's there's that that kind of big thick yellow part which at first I was thinking well that's probably just like a burst of energy that it uses to kind of power up the pen give the pen enough electricity to actually run and respond with the rest of the signal that might be part of it but I think that's also used for locating the pen so I think that that first pulse is the tablet sending some you know basically the tablet is a one

side of a transformer like an electrical transformer and the pen is the other half and the tablet transmits a little bit of energy and then just like an RFID card the pen can kind of modulate a little bit of data back in its response by kind of adjusting how much power it draws from that system so in this case that first burst of energy is kind of locating the pen but then it's actually sitting back a bunch of ones and zeroes and so that was that was an interesting thing to dig into also so I'd been looking at a couple of different tablets as you can tell by me just confusing one tablet with a different tablet but then

I started like going a lot deeper into this particular one it's the Wacom the model number is CTE 450 but it was also marketed as the bamboo fun tablet I liked this one because this was like the first affordable tablet I found this was the first time I could just go down to Fry's Electronics and buy a tablet for a 90 bucks and then much later I found well these are actually really cheap on eBay so I could just buy a bunch of them for like $4 each and then have a great hacking platform and even better actually these tablets came with not just a pen but this little mouse thing and the mouse thing was

actually so much larger than the pen that they were using discrete logic chips so each they didn't even have any custom silicon they were using like individual logic gates you know flip-flops and and gates and things on the board so this was a great project to show you know when you don't have that complexity explosion that comes from software or a system on chip when you actually do just have the hardware out there on the circuit board it can it can totally be worthwhile to actually make a schematic and then and then use that to to get some more information about you know in this case the schematic was made it very clear how the protocol works you could see exactly

where each of the bits are being calculated and then fed into the shift register so the mouse the mouse was fun and easy I did a video episode about the mouse and that didn't take too long but then the tablet itself was this huge rabbit hole so the tablet the tablet doesn't even use a processor that I was familiar with before I I like posted a picture of this processor on Twitter and was asking it has anybody seen one of these and actually the more from Adafruit recognized it as this thing that used to be made by sanyo and then got bought up by own semi called the LC 87 which I guess is been used in like

appliances and stuff it's a very strange chip and there are a lot of different variants of it that all have different peripherals and most of those aren't documented and so it actually turns out I'm pretty sure this is a custom chip for walk on that that sanyo is /on semi made with Wacom specific peripherals on it and it looked like there was a debug interface broken out you know going back to the chisel you know you really want to find like well where is where is a great place to stick this in so I can break the system open and if you can find firmware that's great if you can find a debug interface that

lets you slurp out the firmware then that's that's awesome a lot of times you don't get that lucky though like the debug interface would be turned off you'll have flash readout protection in this case I think the debug interface was turned off and I just didn't even have access to the debug dongle I tried finding actually eventually one of my fans actually sent me one of dongles and so there'll be a follow-up where I actually dig into that but when I was doing this that was unobtainium and so I I just had kind of had to figure out how this chip worked without hardly any documentation but that's what I started following a hunch so I wanted

to read the firmware out of this chip in order to get an idea of what the tablet was actually doing and the I think I think we got a little more in this video yeah the processor was such a problem but I had this hunch where we're already getting a little bit of that info from the processor in the form of those USB descriptors where it tells us like okay I just plugged in my tablet here's the report descriptor here's how all that stuff maps into a packet here's my device string all that and that's in the same memory as the code that we're trying to extract so I was thinking well maybe maybe this is a glitching target

maybe I can just try to interrupt the processor right when it's deciding how long that response should be and get it to calculate that result incorrectly and just send me a much longer packet and just read the firmware out so I started following that some of you might be familiar with this awesome hacking tool called the chip whisperer made by Colin O'Flynn this is an add-on I made for it called the face whisperer because it's also based on the face dancer it's basically the the face dancer is a kind of a saw like a controllable USB host or you can tell it you know I want to pretend to be this kind of device I want to talk to this

kind of device uses this chip the max 3421 II in this case I took that same chip and kind of wired it up in a way that let me send out a request you know ask the device for its descriptors and then get a timing trigger that's cycle accurate to when the device would be seeing that USB packet and then I can use that to actually deliver the glitch just the right spot so this is this is the money shot right here so that that's the power power trace on the right and you can see the the big spike is the glitch and then it's kind of misbehaving for a while and the red packets are

stuff where it just displays the wrong and give me the wrong stuff the green stuff our packets that are way longer than they're supposed to be yeah it starts out just giving me regular descriptors and then you can start to see some bitters show up when it gets close to the interesting part and then and then just all of a sudden huge packets and so that was the firmware so this was this was great I had a lot of fun with this and this seems like a technique that might be applicable to a lot of devices so you might have actually heard about this in Kate and Dominic's talk yesterday because there they're working on a tool

that's like this but a lot better all right the most recent thing that I've been reverse engineering is this gimbal and this is another one of those reverse engineering for interoperability kind of opportunities so I got this is this is for a project where I'm just building a robot that lets my cat hang out with people on the internet it's really super good but it's great I my cats kind of the co-star on these these streams and I've currently got a tripod that I fell over him around with and it just seemed like well we could just put this on a Roomba or something and have it wander around and then people could control it

and that idea evolved into I want to have this camera on wires that we can like follow the cat with computer vision and have people like control it with twitch and and all that so then that led to wanting a gimbal that I can both stream HD video out of and like uncompressed you know 1080p real-time and something that was kind of small and lightweight and inexpensive all the things you'd usually want from a gimbal so I didn't want to get this like huge thing that was designed for a DSLR that already had all these controls I ended up just kind of picking this small thing designed for quad copters and then hacking on it until it worked so part of

that was mechanical modifications and trying to make the right brackets to accept the camera that I wanted to use and putting some more signal wires through the joints in the gimbal but a lot of it actually turned out to be software like a lot more than I expected we could have just controlled this gimbal the way it was designed you know reading this reading the side of the Lego brick and just connecting it the way we're supposed to and and that was that was what I tried at first but there was something missing I needed to know where it was currently pointed like you can give it do stick inputs and that'll move it left or right or up or down

then I have no way of knowing well where does it think it's pointed right now and I would have had to add a separate sensor for that which would have been super annoying so then I started to take it apart and try to tie into the existing sensor electrically and then somebody kind of nerd snipes me with like why don't you just pack the firmware or why don't you just like talk to the serial port and I'm like well this seems easier but then they eventually convinced me and I started to dig into the firmware so we found a debug port it was you know like armed Sarila wire debug but flash readout was disabled

it had firmware updates but the firmware updates were encrypted so it didn't look like there was a super easy starting point but there was this it came with this little tool that you could run in Windows it would connect to the gimble over serial port it kind of repurposed the pitch and yaw controls as like serial Rx and TX but only if you responded to this magic packet right when the gimbal turns on so you can get an idea of how this works by just reverse engineering the existing Windows apps you know you can either take apart the binaries but in this case I was actually kind of doing it at the protocol level and so I looked at the

serial packets I used Sig rock and started making a stack of decoders and Stiga rock is an open-source framework for logic analyzers and it works with a lot of different hardware I was using it with one of the older salient logic boards but in this case it was really handy because you can just capture a big data stream and then start writing Python programs on top of that to just process the data so for example I would end up with these traces like yeah we were like trying to work out the CRC's together on the stream and we just ended up with this this gradually higher level view of what was going on by by first looking at

the serial data and then just trying to figure out more of that and add more decoders to the stock so that was that got us pretty far

you know the the gimbal project there were a lot of wrong turns in that so the serial protocol it turns out was kind of buggy and there was a way that you could just send it a packet and well there were multiple multiple memory corruption vulnerabilities that were just really really stupidly obvious like there was one where it's parsing out packets and the packets can be up to 255 bytes long because it's an 8 bit length field and they're only storing it like a 32 byte buffer you know just stupid things like that and this is this is still all over embedded systems as you might know from if you've attended any other talks about

the security of embedded systems so actually this would this turned out to be kind of a hazard because I ended up kind of breaking the gimble by sending it a bad packet and so I actually over wrote the settings that told the gimble like where's the motor calibration and where's the like center of the magnetic encoders and so at that point you just power on the gimbal and it just immediately tries to break itself just like physically slams to all the end stops and no good so then there was kind of a little sub task of how do we get back out of that and that developed a lot more that needed a lot more tool

work and kind of understanding of this protocol so just still building from that serial protocol I ended up kind of getting a little higher level and trying to see well what are all these different what are all these different values mean I found this kind of shared memory space that all the different control loops in the gimbal used and there were actually multiple separate microcontrollers each joint in the gimbal at a separate micro and and so it was just an interesting reversing journey and you're trying to figure out what are all these settings how do I fix the thing that I broke ended up getting a second gimbal donated and that one actually was kind of the

known good one that we would transferred some settings from but of course they're different gimbals and they have different calibration so then we had to figure out how do you calibrate them so that was that was a nice rabbit hole and I tried to document that as much as I could and so I eventually end up with something that kind of works this is the view from the camera and this was a little unstable but it was actually not just like wildly oscillating back and forth anymore which is not I was kinda hoping to avoid having to dive into the binary for this project because it just seemed like you could get everything you needed from that

serial protocol but because because I needed to figure out the calibration and because there was just so much going on in there it didn't turn out to be really useful to take a look at the binary and this was something that we got access to kind of by luck so I mentioned the binary was encrypted in the firmware updates and the the the the board like each micro controller has a debug port broken out but you can connect to the debug port but you can't read flash memory as soon as you attach a debugger it disables flash read and so if you try to read from flash if you try to run any instructions that read from flash it

just holds but you can still read SRAM and one of the one of the folks in the stream actually just happened to notice hey this SRAM dump you posted there's a thing in here that looks like a private like a encryption key looks like here's the AES key schedule and this might be a key and just like tried it out and yep sure enough we can decrypt the firmware with that it was just a little bit of guessing about what modes to use and figuring out what the initialization vectors were oh yeah yeah I think I think we actually originally found the key just by brute forcing it scanning through the memory and then and then and

then after we found the key it was like very obvious that here's the rest of the kind of leavings of the AES algorithm in the bootloader you mean you learned so much about the simple problems that you can avoid if you're trying to make a secure system anytime you take apart someone else's system like just just clear the keys I mean not the way Wi-Fi is clearing the keys but yeah yeah there are there problems with that both ways huh so yeah I mean they're there turned out to be a pretty deep rabbit hole in this particular reverse engineering project and this one isn't even really done I've got this to the point where it's kind of

usable as a you know as a camera gimbal but there's still a lot more work to figure out how to tune it so it doesn't wobble as much and how to just get more information about what each of these parameters actually does and so I'll be I'll be continuing the live streaming on that and doing some more reverse engineering on that and trying to share as much of that as I can yeah so that I mean I think that's about as much as I'm going to talk about the gimbal right now but it's it's still it's still quite a fun project I mean I think I don't worry where it leaves me though cuz I feel

like as I mentioned at the beginning of this talk I think the way we have been approaching this is a little bit wrong sometimes we think like because I get this from people I talk to a lot like they see the stuff that I've like edited into a super condensed video and then I post this stuff and people think like oh you can just do all this stuff instantly can I just give you some stuff and you can reverse-engineer it all like right now and I try to tell people actually it's not it doesn't really take that much like skill I mean obviously skill helps obviously when you do this stuff you learn how to do it more efficiently

and do it better in the stuff builds but I don't think that's the primary aspect of this the primary aspect is just investing a lot of time and being really patient with the technology and like knowing that all the stuff that you're working with is like piles have melted Lego blocks put together by people who are really overworked and tired and and it's it's much more about about that just understanding the just how deep that rabbit hole goes and so that's why I've been trying to share not just the really condense stuff but also like what does this actually take trying to do this stuff with people in real time and sharing it on the internet it's also

trying to get more of that in person and I think I think that's a way that we can kind of counteract this vision of what hacking might be like and kind of push back against this idea like why why why are we even spending time doing it computers should just be easy for like people who are really good at computers it's like well no even when you're really good at this stuff it's still hard and it still takes a lot of time so I think I think these aren't necessarily the kind of hackers I want to hang out with or learn from but they make great TV characters I think this is maybe a bit more the kind of hacking

that I that I feel at home with so you know I feel like it's more like you know craftsmanship or like woodworking or we're like scribes like copying manuscripts I feel like it's something that you know it should be treated as this this this activity where it's not like you've achieved a thing and you have this status symbol of like I am a hacker it's more like I do I have this ritual of investing time in this particular practice that can can get you closer to understanding of what you're working with and so I think in that way it is it is more like you know like studying a book or like doing science I think a lot of

what we do should be thought of a lot more like the physical sciences because the things that we that we're trying to understand now they're so complex that you can't just ask all the people who worked on it you know you've got to take an approach more like we use for physics or biology and find a way to conduct experiments and actually figure out for yourself what's going on inside something so I I would encourage everyone to just try to find something that you want to take apart and focus a lot more on finding something that you're interested in then something that's like particularly like hard or has a particular technology in it or anything because so much of it is just

about the actual time investment that you make and if you can find something that'll just just really get you obsessed then the rest of it will just fall out you know you'll you'll figure out the problems one by one using the same process that you're approaching the whole project for like that's the cool thing about reverse engineering is not always the most efficient way to find an answer in fact it's almost never the most efficient way to find an answer but it almost always works if you're willing to and you have the time to put in so I think that's that's what I'd like to leave you all with today yeah happy hacking

you

you