Developing a Hardware Training Exercise for Defcon IoTVillage - Deral Heiland

thanks so much for everyone coming so uh it'll be online I'll get on the stage my kind of knees bother me I like to wander around and so if I go up and down the stairs I'm sure I'll fall and break my neck so today's talk is on developing some iot Hands-On exercises that I do for a Defcon every year in the iot village I've been doing this for a number of years uh so I'm hoping you enjoy this you get something from it at the end I'm going to do a a recorded demo of this year's exercise and kind of walk through that for you at the end I was planning on doing it live but uh my virtual environment is kind of like wonky uh so I thought that would go really bad so let's go ahead and record the whole thing and then step through it explain the various steps so everyone learns from that so my name is I've been looking at a rapid seven for uh come up on 10 years my current role there is I am a principal security researcher for OT iot what does that mean so I do anything related to OT and iot at rapid7 I function as a manager I get a very small team of really cool people and I help pin test teams developing the new testing methodologies and stuff like that around iot so let's go the first five years so like I said I've been doing this for five years and the exercises we've done cover up a large Gambit so 2018 we kind of went through and started teaching people how to do we did multiple exercises how to use a logic analyzer how to interact with uh you are a universal asynchronous receipt transmit how to do um de solder we actually tried to do that one that was a total fail because obviously trying to teach people how to do complex D soldering in a short exercise uh ultimately turned into a mini train wreck for the most part and then the last one was like good processors so we had an exercise around pick processors going from there the next year uh let's go ahead and jump to the next slide so we can go into some of this stuff more detail like I said the first year we had all these exercises like four of them and everyone had a chance to walk through those uh the following year we did exercises four different exercises on and developed these with exercise manuals the whole nine yards to pull firmware from embedded devices and we covered multiple ones how to do software uh swd how to do CJ tag which is a subset of JTAG how to do spi how to do any circuit serial programmer and we had four different devices we had four different debuggers that we had them do this with so they got our hands across a whole lot of different tools a whole lot of different devices uh even one of the pictures up there we actually had one of the wristbands uh so it was opened up so they could interact with uh um steroid debug to actually pull the firmware off the device the following year we got into an attack against the device so I started ramping it up I'm like look I've shown you how to use the tools how to do firmware let's kind of let's kind of ramp it up so we wanted to do an exploit how do we break into a device to get root level access this particular device is a mesh Wi-Fi in this case here we wanted to get into like uh the yearbook console but the problem is you couldn't get to it it was for the most part there's a um a setting an environment variable if it's set to zero it will not give them a chance to break out into that so we did a thing called ping glitch in the pin glitch that would actually between um Uber loading and Ubi loading a kernel there is a fine few milliseconds that if you ground out the communication line to the actual Flash the device will drop you into a uber console so we showed them how to do that to piglet just by shorting this out during the boot process to gain access and then from there makes alterations to eventually gain full access to the device the following year I think was 20 20 20. I missed that one uh Defcon was canceled for real which is kind of interesting but we did a bunch of stuff online so we kind of circled back and did things online covering the first Year's exercises logic analyzers things like that so then we went on to this year so I'm going to walk through the whole process to how you get these exercises developed up in this particular case we wanted to do something different we wanted to build on previous years also wanted to have it so they can get kind of root level access to an actual device again but I wanted something new so I started thinking through this what would we want to be able to do new what I wanted to do was hey can we do a memory style attack so I started doing uh research okay how would this take place my first initial attempt was I wanted to do JTAG the concept was within JTAG I wanted to be able to interact with the device and alter its memory to execute some kind of code or bypass some kind of setting to gain root level access so I kind of start that whole process of finding devices that fit this model and it follows that Loop research by tests throw it in the trash and start over and this was like really crazy bad last year the step back to last year trying to avoid that the previous year to actually gain access to the device that would involve D reading settings changing settings on the device the particular device we had that right earlier was a modem in that particular case in another year we don't want to make the attack something you have to bring with you other than Hardware testing we wanted you to be able to actually attack this system using the software on it since it was a cable modem we wanted to be able to boot the thing up break into it make alterations to the system so we actually connected in using an SD card breakout into the device in that particular case it didn't work I allowed to that went through the whole thing and it failed everything was working but when we got to the basically the root console the device we had no prompt it didn't work so I contacted tons of people that are in the whole cable modem hacking community on how to do this and they literally told me you can't do that you have to install your own firmware to accomplish that that's how we always do it I'm like well that's not doable this is something we I what do we learn just by putting our own firmware on the device we want to break into it and get access to the existing vendor-driven firmware so we can do other testing attacks and things like that so what was happening was I was getting to like a SSH connection and all the settings and everything was saying you have a valid SSH connection but I never had a problem well apparently when SSH and I can't remember the switch right now you can actually alter the ssh's environment structure what was happening on advice the vendor had put all these Protections in place one of them is they created an environment that if you enable this on the device to get root level access they would set an environment variable that had no prompt so even though we got to the system you couldn't get a functioning prompt so I'm like two weeks away from Defcon going oh hell I'm in trouble worked all day one day and that friend of mine goes hey there's this SSH environment variable set that you can actually set your own environment I'm like let's try that I did that it failed and I'm like one more test and then I'm gonna panic and try to find another solution for Defcon well it turned out that if I set that variable not to have my own environment but to disable all environment structures it worked I got a blank screen but it took commands and executed them at root level so we didn't get a typical prompt where we got it to work so I wanted to avoid that this year so I started early through this whole test process and we went through a number of different devices up and left-hand Corner we looked at a ring home alarm system took it apart got into it hacked into it found out they're a disabled old JTAG on it so I took that a hundred dollars worth of gear through the trash moved on the next device which was some kind of smart plugs same way it had functioning stock but there was no data sheet so one of the biggest problems we get into OEM devices so there's no data sheets for the processor so then I got down here to that Ott box in the left team corner and then I had said in my lab because I always try to use do you have City on my lab to avoid spending unnecessary money so I got into that device and I'm like cool it's running a 905 processor it has a functioning JTAG cool our laborities on the market so I go out with that device and find out none there's an environment Market eBay there's no used ones on eBay on my comp screwed so I go out and find another vendor that produces a similar device with a 905 processor and I order several of them because I think I got the right device I'll be able to figure this out so I'm bringing in about a hundred dollars worth of gear I open it up and it's a 905 processor but it's a 905d processor 9905b processor turns out that processor basically is totally different pin out I think it had like 48 pins versus 24 pins why the hell are they called it on 905 I don't know it wasn't even the same process or in any way shape or form and there was no data sheet for anywhere on the market they had OEM that processor so again it was like okay I'm not able to spend days or weeks trying to figure this out I need a functioning uh JTAG and I need one of the data sheet threw that in the trash and during this whole process I'd also that camera sit in the middle I had found that camera had played around with it really no JTAG the processor didn't have any data sheet but I did find something that was interesting in it so that was a kind of a memory alteration type attack that was published online so I utilized some other people's research in this so I set it aside and I continued down the JTAG so when I hit that demarcation point it's like you find a solution or you're gonna have to go back so tested another device the wireless smart giveaway same problem there we get the device we take it apart looks like a good candidate turns out that jtags disabled on this processor we can't go in and re-enable it or any of those type of things so that's the way I kind of back off I'll go okay let's go ahead and dig into the actual camera and try to develop an entire test process for that so we take the thing apart yeah it has a simple memory example and it's cheap this is cool this is like 15 dollars so I immediately bought a whole freaking shipping crane of these things like 12 of them or whatever and I'm shipped to my house and it cost me 140 bucks I'm 50 bucks like I said cheap they're like 15 a piece on these and I took them apart and I started whacking way at him but okay we need to develop an entire process where we could go through this thing and teach people so we have to add other pieces to the puzzle more than what was actually published online all right so we take it apart it is cables inside of it whip all that off we don't need all the extraneous gear the main thing we're after to get re-level access to this in a test environment is we want to actually have a device where the CPU is running memory is running and we can interact with it in some fashion so we can disconnect almost everything so we get into doing that and we start looking at the device so we have a CPU um this is uh says 38 wx1 this happens to be xm530 processor very common processor used in very cheap chinese-made Hardware out there so that's good now the cool thing it has a SD card on this device that means if we want to get firmware off the device in a training environment we can simplify it we can teach people how to read it off the flash and eventually write it out to the SD card as part of the process and then from there we can pull it over our operating system do whatever we need to do to it yeah we also it turns out it has a 1.27 millimeter Heather I don't know if anyone here does any hardware hacking type stuff in my lab I have the tools for everything no matter what the header is a device I have ways to doing that but that has its own problems it's small it's very small it's difficult most test leads don't really attach well to that header most test leads are often designed to better connect to 2.54 millimeter headers and then have a Wi-Fi didn't really need to utilize the Wi-Fi but I wanted to point it out the Wi-Fi device in there this is kind of cool I've been looking at a lot of these cameras and and not to divulge off onto something else but again I always do research and thinking of new ideas and ways this device is actually communicated too from the camera structure processors on this device uh using USB so it's in circuit USB communication which is something I'm looking at into it's really fascinating if you ever dug into the USB standards or how USBS are laid out on the circuit board it's insane high-speed USB is so impacted by noise and Communications and stuff like that especially in circuit communication how they Define these on the circuit board is amazing the runs and the ground lines have to be certain micro millimeters apart and all this type of stuff or the communication won't work so in the future one of the things I'm working on is how to interact with internship Communications on USB in circuit so hopefully we'll see something come up in the next couple years on on how to deal with in circuit high-speed USB Communications for intercepting because you can't draw a standard logic analyzer on this thing and capture it because it's such high speed most of them don't run at that speed uh and then we have flash memory so uh this one's nice uh if you want to get the flash memory off this thing you can always just uh pull it right off the chip but I don't want to make that part of the exercise that can become problematic if you're dealing with uh an i2c communication ship it's fairly easy just to piggyback a clip on there and read it because the interaction with that is channel based so most of the readers will actually uh Advocate the channel for reading the data but if it's SPI you run into a problem with piggybacking sometimes because the actual CPU will keep grabbing the SPI from you and you have to set the CPU to reset sometimes to actually be able to read that and then you know we are alternative is to actually pull the chip and I'll be honest with you if I'm working on this device here myself in my lab I've got to play with my solder D solder skills I can desolder that ship pull it off and read it and put back on less than 10 or 15 minutes I've gotten efficient at it sometimes that's easier than trying to track down a JTAG or deal with other methods I can often just desolder the chip and read it but for training purposes that's really not realistic either one of those ads some level of complexity and our exercises we want those to land within a 15 to 30 minute time frame we set up multiple stations at Defcon and we run hundreds and hundreds and hundreds of people through these exercises so if it starts getting dragged out to an hour and we can't get as many people through the exercises we want to educate and train and give people that experience of interacting with iot Hardware so we had to take that out and look at other methods as we get into oh well let me go back here real quick uh so also the other chips you see the lower one down there and one slightly to the left and the side right picture one of those is a it's kind of a weird motor controller even though there's no motor control stuff on here so I think it's being used as an 80d conversion chip is how that's function and the other one is associated with some USB uh Communications coming out of the device okay so uh so how do we deal with the 1.27 millimeter in training I always like to make things simple well this third year for everything out there so we just kind of go out and uh we buy one so it's a 1.27 to 2.5 millimeter this makes life so much easier these things are cheap you buy them a box of like 10 or buck two piece so I like I have a crap load of these in my lab for here I don't want to deal with this I just solder this on there and take care of that problem so we solder that on there there's four leads you can see four leads soldered up there so you have uh your communication here ground Trend transmission receive the one that's not connected with voltage in this case it would be 3.3 volts and you really don't need that to actually interact with the device just ground transmit and receive is actually needed in the process so we have to build the exercises what's involved in that so once we've developed all of the main steps of the exercise we have to build Hardware so I have I have the I'm looking at the picture and I have a tendency but that these conferences I like people to be able to get a closer look at it so here's a couple of cameras I'll pass these around uh and here's the actual device uh test boards we set up I do this every year uh the cool thing is this year I thought hey man let's like it is Defcon let's put it on black acrylic the worst freaking thing you could ever do you know we're all like paranoid of disease and infection this stuff shows every piece of dust every fingerprint you know if some of you would hear the cost is going to show up and you're thinking oh what was your thinking Daryl when you put this thing together so I'm sure you're not going to get anything from it because I'm like disease free but um here you go and then here's a couple of camera devices you look at as you pass these things around take a good look at them well I asked you to be careful with them uh that device will be used at Defcon so when I get out of here I got to go back and retest these make sure they work so try not to drop them bang them uh Fry chips off it with a pair of pliers in the back of the room uh anything like that so from there what do we need to do multiple things we need to board these that's wrong with these so I go through buy all the gear for this and assemble all these things in my lap so we've got that built the goal of building these things is we want it easy for the user to interact with and we want it easy not for them to destroy the devices so we found out Mountain among these things is pretty good so we'll see how that turns out this year last year I was terrified the exercise was complex because when we were putting the SD card connection into the device where we tap directly into memory on the embedded multimedia controller with SD we backfed power from the SD card reader into the device it would power up the multimedia controller in the flash memory and that's all it were powered up that was always used the way the circle was designed it allowed us to do that the problem is if you had that plugged in and the person doing the test Plug Power into the unit at the same time the smoke got out so there was no physical smoke but it destroyed the device okay on top of that we also had them pulling memory off alternate they DD it off the device they would alter it repack it DD it back onto the flash of the device so what could go wrong there right uh the fear there was while we're doing this you were often they're often communicating over Dev SDC which was the uh the SD card reader on the device that we were bringing up but if they put in SDA they could live over without the hard drive on the laptop with flash memory from the device so I'm thinking will we survive Defcon this year we actually made it through all of that with with 10 devices we only lost one device we had one gentleman who's like hey this exercise is not working I went through everything I walk up and he has both power and the USB plugged in or the SD card reader plugged in I'm like first you've burned the device out on top of that he overwrote the hard drive with the flash so the same guy did this so so you know you're thinking this guy works for somebody I feel sorry for his employer because the exercise w