Assessing the Embedded Devices On Your Network

- all right David Thomas check you on track - assessing the embedded devices on your network thank you very much to our thank you so a little bit of Liggett ory about I'm a security engineer on the security assessments team at Google security assessments basically means I get to break things and get to break a wide variety of things I also play a little CTF actually I'm also one of the organizers of the B side CTF so if you haven't checked it out this weekend you should I've got a bunch of certificates but for anyone who hasn't done them the offensive security certificates I absolutely love those because they're not just multiple-choice tests so those

are two that I actually care about putting on my resume so insecurity of things right thing internet of things and security things about the same so I came up with this great idea they're like hey we've got this wonderful technology called the Internet and we have all these things that I love to control and it would be so much cheaper if I could control it from the internet and all of this was developed at exactly the same time as we're like seeing vendors get popped left and right we're seeing target breaches Home Depot all of these things so of course people like let's connect everything we can to the internet what could possibly go wrong

so why do you care about this right so I do a lot of black box assessment and there's several reasons to care about it one of them is determining whether or not a given product is a product that you want to be using in your environment and while a black box assessment isn't going to find every vulnerability right even a white box assessment might not find everyone you can usually get an idea of the security posture of the product and of the vendor that you're dealing with from a black box assessment if it takes you five minutes to find the first vulnerability you've got to have a bad time on the other hand some of you

may be working red teaming or pen testing how many blending your new red teaming or pentesting professionally yeah I'm gonna tell you a little bit about why you should love Internet of Things devices on the network that you're going into there's a lot of reasons and finally you get this opportunity to find a bunch of low-hanging fruit these devices are basically like honing 90 style so the target audience you know if you know embedded hardware and internet of things inside and out I'm probably not gonna tell you a whole lot new but if all you ever look at are software applications and you've never touched hardware then hopefully you'll come away from this with an opportunity to break down some

of the hardware that's on your network and I believe me all of you have some of these devices on your network even if you don't know about it even if your security policy says you don't have any of these devices on your network so things will look a little different from your typical assessment right most of the time if you're a pin test or something that you're looking at Windows domains and Windows boxes and workstations and you can get your foothold by client-side malware you can get your foothold by phishing those are not the techniques that you're going to be using here the techniques that you're going to be using here are extremely different from that almost all of these

devices do not run Windows there's an embedded version of Windows and I've found a few devices that have like Windows XP embedded which I don't know which is worse Windows Embedded or Windows XP but either way you're now full of holes but don't worry those only work on safety critical systems there's but so most of the time it's Linux on most of these devices you'll find other operating systems there's like free RTOS which is a real-time operating system there's a cost there's also VxWorks has anyone ever seen a VxWorks device before does anyone use the router provided by their ISP 50-50 chance you have a vxworks box sitting at home almost all of these devices do not use x86 CPUs

until very recently x86 CPUs were incredible power-hungry right atom processors have gotten a little bit better in that space but still they're some of the more power hungry devices so if it needs to be power limited or thermally limited or cost limited in a lot of cases they're using an on x86 processor and most of the ones that I've seen have been based on arm MIPS is a close second cousin I recently did an assessment of something that had an embedded PowerPC processor in it so I got to learn coding on PowerPC for the first time which is kind of cool if you have the time to do it and you usually have almost no

visibility into what's going inside you don't have a user interface like you do when you're looking at an application that runs on a desktop or a web interface usually the user interfaces are really really really weird interfaces like you might have a little touchscreen but more often than not some of these devices you have like some LEDs and buttons and that's your user interface so good luck figuring out what's going on under the cover from that and finally many many many of these devices are not built by companies that understand software like if you look at the things that are out there a ton of these devices and not the stuff that comes from Internet of Things startups

but like industrial control systems they're made by industrial control systems companies their customers came to them and said hey we want a network feature so the industrial control systems company either went online and was like alright we're an industrial control system company what kind of software engineer can we hire and who wants to work here well turn adoulin they took one of their engineers gave them learn C in 24 hours and said guess what you're the network guy now either way it's not the highest quality of software that you'll find they have typically very bad software development practices and very often when I report issues to these companies I go hey I need to reach a security engineer or

security manager or something like that and on more than one occasion I've been routed to the guy who's responsible for physical security at their facilities so these are not companies that are really really really on top of the networking game and so I don't blame the companies entirely so everyone should be familiar with the the CIA triad of confidentiality integrity and availability right this is how you begin your threat modeling for any assessment and while these have many of the usual things that you'd look at at any in software application they also have a variety of unusual things you have data that you don't expect to be on there if the device has a microphone

embedded in it you can't assume that the only audio that that device is exposed to is when someone is intentionally recording with it you have to assume that any audio that is within range of that device would potentially be compromised by this likewise some of these things have really really really strange constraints about the integrity how many of you would love to know which elevators you ride in are connected to the Internet yeah they are also industrial machinery and industrial plants I mean how many of you have seen reports of power grids getting hacked right we're not just talking about like integrity here doesn't just mean there's some data compromised integrity here means people lose power people get

injured people could potentially lose their lives from a lot of these embedded systems so integrity suddenly becomes a really big player there's no going back on that you can't restore from backup once someone has been injured and again availability is a big thing of that right like what if you found a bug that was just a denial of service and an elevator controller it's still pretty traumatizing to lock someone in an elevator until the elevator repairman gets there so I want to go through this and talk about the techniques that I've used to assess things using a case study this is a voice over IP phone it should be pretty reachable most of you probably

have one of these maybe not this particular Graham but have probably seen some of these in your environment and it's still a network connected embedded device right if you're doing an assessment the first step is always to do some recon and it turns out that anything that has a wireless chip inside of it in the the sold for commercial sale in the United States has to be approved by the FCC and part of that approval process is to the sticker on the bottom that has the FCC ID on it which you can see on this sticker in the bottom right corner and then you can go look up things about it like what bands it uses and other useful

information there's a wonderful website FCC ID I oh it's not the official FCC website the official FCC website is pretty bad this site made much much much more usable there's other ways that you can find out what's going on read the user I know we all hate reading manuals but it turns out you find surprising things when you read the user manual the mistake of not doing this in an assessment once and we were looking at it for a while and then I get to the page where it says if you've lost your password you can use this username and password to reset the passwords on the device hard-coded trance in the user manual way to go also grab the firmware

from the website if they provide firmware and you can download it you can grab it from the website you can do all kinds of things with the firmware right there you have the files that are on there so many of you probably are familiar was encrypted firmware and the possibility that you could do that and encrypt firmware before it's downloaded to make it harder to reverse engineer anyone care to take a guess how many times I've found encrypted firmware for a device yeah I already said zero that's correct nobody encrypts their firmware so this is from the FCC database this is a picture of that same phone from the inside you can't see it very well on the

slides here but you can actually blow this up and find out exactly what CPU it uses inside it's an arm CPU and you can find a bunch of other use of potentially useful things off of it you can find the model of flash that it uses the model of RAM that it uses you know how much RAM and flash it is what's interesting about knowing how much RAM in flash a particular device has is you can narrow down the operating systems that you might be looking at if it has like two megabytes of RAM it's not even Linux it's maybe running the exports or free rtos or something like that so firmware there's a wonderful tool out

there called bin wok and what that does is it goes through a binary blob and extracts other known file types from within that binary blob and it gets you all of these files out of it and you can use it and it'll even find compressed files that are inside that so like the firmware might be a GZ with it with some special header on it don't find this extract the gzip and give you the raw file and then I love to just run strains on it right I mean strains is a wonderful tool for getting your your feet wet and getting your orientation to it and so I'm doing this and I've probably been looking at this device at

this point for a total of two hours and I run strings on the firmware and I find these strains here and I don't know about anyone else but if you've ever seen command injection this is what command injection looks like they're like oh I don't want to actually make my program talk directly to the Wi-Fi interface that's kind of hard I'm just gonna use the command-line tools to do it and we're just gonna pass parameters on the command line feel like sprint up style strings so yeah that's it that's a good place to start looking so they like to go live right and so this is that target board I've taken the back off of

it because reasons that we'll get to in a moment and I plug it into my laptop and I actually use two USB network interfaces and put a VM on my laptop to man-in-the-middle all of the traffic coming off of a device because then you can see everything that's going through it even if you don't actively man in the middle SSL you then know what hosts is connecting to you know whether it's reaching out and downloading firmware on first boot you know whether it's reaching out and grabbing config on first food it's really useful to do this like from straight out of the box do it on first boot because you can see a number of steps that it may never repeat

the reason I have the second network interface and I'm not just depending on it by the way I didn't want to put the other end of this connection on for Platt on to our quark network so the other one goes into a lab Network instead of going out to our Corp Network so if you have a suspicious device you may want to consider doing something like that if you have a similar setup and so isolates traffic from your testing goes when you're in your VM right so the boot up like I said is very interesting maybe you want to man-in-the-middle gtp or HTTPS or sweet anyone it's wonderful for that it's also interesting when you start seeing all

the HTTP traffic flowing and you realize you didn't install a certificate on the device they're not verifying certificates then I like to use n map right and map is one of my favorite tools in my toolbox anytime and I scan all the ports this is sitting one network table away from my device I'm not worried about traffic I'm not worried much about dropped packets that's why it's t4 and not t5 and turn everything on and maybe you want to turn on any map script I get an interesting experience at this particular device thanks to n Maps tricks the first time I ran an map on it sitting there waiting for the results to come back it takes a

minute or two and about halfway through the scan the device reboots it's like well that seems interesting let me see if I can reproduce that run and map again the device reboots again right now to map again no this isn't a fluke the device is going to reboot every time I run and map so it turns out that I found a crash just by running to end map scripts that triggered the device to reboot the hardest part of all of it was narrowing them and that script was running at the time the device crashed by the way why are sharp helps look that you watch in Wireshark and see what packets are just sent when the device

crash so let's talk a little bit about like the most basic hardware tricks right this particular one was actually super straightforward because on the board then these four pins sticking they're so how many of you have actually used a serial port in your life oh wow that's actually more than I expected there's a lot of people even in this industry that I say serial important they go what so serial port normally has like nine pins on it but it turns out you really only actually need three pins to communicate all you need is ground TX and rx and there's this wonderful adapter you can buy Hardware serial port by the way hardware serial port is 12

volts this board runs at 3.3 volts you put 12 volts into a 3.3 volt board you will let out the magic smoke your assessment will be over until you buy another device but this little cheap USB to UART adapter I think this one's from Adafruit maybe from Sparkfun something like that they're like six seven dollars people use them to program Arduinos so it's a totally useful tool and you can connect it right up to that and you don't ever want to connect the voltage output line you don't want to drive the board from two different voltage sources so you put the board into its own power supply and then you just don't connect the usually red wire as you see it in

this case you also if anyone's ever used a bus fire it you can also use a bus pirate to achieve this but you have to go through like eight levels of menu and the bus pirate to get to this so even though I have a bus pirate and much prefer using this for yours so you know you the pin out though is important to being able to connect it there's a bunch of different ways you can do that so the first pin you wanna identify partly because it's easiest and partly because it's the most important is the ground pin and it's actually really easy you get a multimeter you set it to resistance mode and you touch one of the

probes to anywhere that you know to be ground which is for example this particular device had some USB ports on it the outside of a USB port the metal chassis is always grounded so touch the multimeter probe to that end touch the other multi meter probe to each of your each of the pins that you're investigating and when you find zero as the reading zero resistance you know that you have found a pin that's connected directly to ground so identifying the it's usually straightforward identifying the rest of the pins can be a little tricky so on most devices I've never had one have a problem with this most of the time I'm lazy I try the RX in tx1 way if it

doesn't work I swap the Rx and TX and then it usually works that's the lazy way to find it there's usually not a problem with that they're not sending enough current to cause damage if you have a Mac words but if you want to be proper about it if you want to do it the right way you hook up an oscilloscope or a logic analyzer or something like that and you can actually see the traffic coming across here so at the bottom here I have an open source tool called pulse view using about $20 from China cheap logic analyzer that's just an FPGA basically and I'm actually extracting the data from New York there's two channels there one of them's

completely flat and one of them has rise and fall of signals and that rise and follow the signals tells you that's the one that's transmitting from the board that your is your target essentially and also the logic analyzer is great because it'll help you figure out the Badri write serial ports can be configured at a number of different baud rates and I've seen on embedded devices I've seen 9600 I've seen 19 200 I've seen all the way up to a 115 thousand 200 or whatever it is in terms of bond rates I've never seen faster than that but manufacturers typically just pick and so yeah this is something you can do by trial and error or if you use the

logic analyzer oscilloscope you can actually just see when the rise and fall are occurring and figure out the Barbary Coast view is nice there's a script that will actually take the data from pulse for you calculate the widths of the bits that are going across the network and tell you what the baud rate is you don't even have to do any math or anything like that yourself so this is a particularly useful you are I hooked up to it and let the device boot and I get all kinds of boot messages you can find out a ton about the hardware and I got to a login prompt at the end and I was like well it

doesn't hurt to try I'll guess a few obvious passwords and so I type the root in and it didn't ask me for a password obvious password no password which really isn't the worst thing ever because they don't run an SSH server or anything like that I mean this is entertaining but in and of itself it's not really a vulnerability because if someone has hooked up and you work to the device you're like in your facility you've lost at that point anyway but it's just really amusing sometimes it's super easy like this because then at this point you're like oh I have a root shell I can do whatever I want to investigate this device if you get this

on a device and you don't have a firmware download or something like that you can use DD and read from the raw flash device and you can create your own firmware image from the device which is super useful if you're not able to find a download firmware or something like that the other benefit of having a shell while you're on it is you can find out what happens when you are interacting with the device so at that point I wanted to do some sort of fuzzing or something like that right like a lot of bugs these days are found by fuzzing I'm lazy letting a tool find my bugs for me is in fact the way I like to do it in

a black box assessment and one of the other nice things is fuzzers just sit there and run you can start a buzzer and then I can go back to Ida and do my manual assessment in parallel with the buzzer and be finding bugs at the same time but it turns out like normally I spin up a fuzzer on my workstation or spin it up on a server or something like that pretty straightforward but in this case you can't do that first off it's an arm so I can't run the binary natively on my workstation secondly even if I try to run QEMU or something like that it fails at a bunch of different points where it's

trained to open device drivers for the devices that are built into it and I was like well at this point I have two options i can either figure out how to patch around all the device drivers and then figure out which of the bugs that I find are caused because I just made device drivers for a terminal or it can try to flows it on the device and this is the first time I ever tried to fuzz something live on an embedded device that's like this can't be too hard right so it turns out actually the reason it rebooted every time I scanned it with any map is they install this thing segment II Handler and the seg fault

handler rebooted the device every time it's like Fulton well that sucks every time I caused it to crash the whole device is gonna reboot how am I gonna get any staples of that how do I capture the state of the crash so I can figure out like what the source of the crash is so the first approach that I took I cross-compiled gdb server for arm stuck the gdb server on there and started and attached it to the process and so instead of getting to the seg fault handler JDBC would pop up on my workstation be like oh you have a prompt seg fault was captured you can get a back trace you can inspect the stack all

those nice things but that meant I had to do it in real time I had to sit there with gdb open and like actually do something every single time it crashed and then the device would still reboot as soon as I continued in gdb so I had to wait for the device to boot again and so I was having this manual process and that's not automation like fuzzing is supposed to help you not make your life harder at this point I had made my life harder so we tried a second approach we decided to modify the binary get rid of that segmentation fault handler and use the core pattern support in the Linux kernel anyone familiar with this so the

Linux kernel writes files called core files whenever a program crashes and by default it just names these core and stores them locally this particular device I think actually redirected them to Devon all because you don't really want to store core files on a small embedded flash drive but it turns out there's a cool feature where you can start this file name with a pipe character and you can pipe it to an arbitrary program and so what we did is we piped it to netcat to send the file the core file off to my workstation so I just started collecting core files from this when I was going through it and so in this case it's just

netcat to the to the workstation on like port nine nine nine nine and then on the other end I had a little bash script that ridet really did like and that cat - shell out to a file and then put that in a tight loop so I was doing this and then I got a bunch of core files and it turned out that if you start fuzzing an embedded device you get a lot of core files I let it run over a weekend and came back to find that the disk on my workstation was full now it turns out that doesn't mean that I found like ten thousand bucks right that would be really awesome if you found like ten

thousand separate unique bugs but I'd probably still be trying to categorize and report ten thousand bucks from this assessment so it turns out that basically it was just small variants on the same crash repeatedly and repeatedly or same I should say about six crashes repeatedly so ended up having to write a script is a bunch of them out there that already let you sort through poor files and sort them based on where it crashed and things like that except that it's got a bunch of these are hard-coded to look at a IP or our IP which of the instruction pointer names for x86 and x86 64 and armed does not call its program counter EW

or our ID so some of these scripts categorized all of them is the same because every time they tried to get the value of a IP or our IP nothing so it turned out that I ended up writing my own script I could have fixed the other ones that are out there but at the point I was trying to categorize them it so I was grouping them by where the program counter was and what the stack the stack registers were at that point Aris the stack pointer and base pointer work and so I started grouping them and then doing script analysis of these core files and it ended up actually this fuzzing process was so

relatively fast that I didn't that I was still the slow part of the fuzzing process even though I was doing this on bearable on a CPU that runs at something like 48 megahertz and piping the core files over the network and all of that I was still finding enough results that I was the slow part when I was analyzing the core files so there are some more advanced techniques that you can use JTAG is this really cool thing built into most stands for joint test action groups so a bunch of processors in the early days had like various debugging mechanisms that were inserted into them so that you could do live debugging on the chip but it turns out that these

live debugging mechanisms were all different from every processor to processor sometimes even within the same manufacturer and so they came up with a standard way of doing this and it takes something like five or six pins to do it but you can actually inspect the live state of the processor which means you can even investigate like kernel crashes using this technique downside is finding JTAG pin outs is really hard because even though they standardize the electrical aspects of JTAG they didn't standardize the pin out and so in the connector can be six pins it can be 20 pins sometimes it's little test points on the board you have to be really kind of dedicated or insane to go after JTAG

on something unless you really are a hardware guy but another thing that's actually not too hard is to dump the flash there's little Clips you can get that clip directly to the flash chip and then you can use a tool like the bus pirate to actually dump the contents of the flash memory of the device and so that's another technique you can use to get the firmware off of the device so if you can't find a firmware image and maybe you either can't get a shell via the UART or maybe they've actually set a real password this is another possibility for getting access to the flash memory and then if you you want to be clever about the things

you do it you can modify the firmware extensively and replace files in the firmware and rewrite the firmware and do things like that they give you more insight into what's going on in the device but we had a little bit of a problem when we tried that so I mentioned modifying the vine it turns out that this particular device uses a file system that compresses all of the files which that sounds great you want to use those little flash as possible sounds great I modified the binary I put some nops in in place of actual instructions the binary was exactly the same size byte 4 right and then I tried to write it to the file system and it

says no space left on device so it turns out that the compression mechanism that they use is different when you build the file system as an image initially versus when you're replacing a file in place because when they build the image initially they have a compression algorithm that goes across the blocks of the entire file system and it can compress more efficiently than when you're using a file system and you're just writing to it at the end because then it's only compressing across that one file and so it turned out that you cannot replace a file with a file of exactly the same size on this file system and it also turns out that this

device completely refuses to boot if you try to reboot it without that program in the file system like it doesn't even get to New York so this brings you to a lesson if you're gonna do this and start replacing files in your flash or otherwise modifying that if ice you're using you might want to have more than one of the devices because there's a non chance you're gonna break the device and depending on how long you've been doing this for the chances of that you know at the beginning they're pretty high then you start doing this for a while they go down and then you go to I'm at this I can start soldering directly

to like test points on the motherboard and all kinds of other crazy things and it turns out your failure rate starts to go back up so there's a bunch of different advanced techniques you can take to this like you can go really crazy but if you're not a hardware engineer and you've never done like any sort of hardware things if like oh the idea of using a multimeter at the beginning of the talk was a little scary to you maybe you don't want to be trying to like desolder chips to read them but like the rest of the techniques should be applicable and like I found several bugs in this particular phone using this multiple memory corruptions which are

always cool to find command injection I probably could have found the command injection during just black box testing of the application I would have eventually found it but running strings on the binary said oh there's Wi-Fi things hmm let me try putting commands in the SSID oh look back six and the SSID work done this particular one this one would have been easy to find without any special techniques if they had no cross-site request forgery protection at all no effort towards cross-site request forgery and they also implemented HTTP digest authentication except it's kind of like they did it by looking at what the absolute minimum is to get a browser to interact with the device and not

looking at the spec or anything like that so the spec says you're supposed to do a bunch of different things for digest authentication you're supposed to generate a nonce it's like all nonce is used only once then you're supposed to verify that the response that comes back has the same knots as the one that you sent you're supposed to verify that the URS you URI and the signature matches the URI that's actually being requested of the website and all these other things now this basically broke down to about like HTTP we playin authentication because none of that was checked none of the extra features it was basically take the md5 sum of the username and password

and send em it was approximately that um so you can find a bunch of different bugs this way it's just the one device so at this point I think I've got like about a little less than five minutes left I could take a few questions and I'm on Twitter and I have a blog that I occasionally post security relevant things to you yes the question was what is the most bizarre hardware device I've ever had to work on and these single most bizarre one is one I can't talk about there's there several projects I've worked on that I either can't talk about right now or can't talk about for a period of 90 days per disclosure

policy but most bizarre devices I've worked on I will say they were industrial control systems of some sort it turns out industrial control systems guys they really are just like oh just building separate networks that's air-gapped which okay it's not the worst thing you can say about Stuxnet kind of proved that even in a highly secure environment with an air-gap network you're still going to get malware and secondly like we generally like no one actually wants to air-gap they're never because that takes away 95% of the convenience of networking in the first place any other questions oh sorry

don't know the question was there's any off-the-shelf software that you can use unlike your home network to c4 known vulnerabilities I don't know anything specifically that's related to like IOT devices you can get like the free version of Nessus like open paas which will let you you can scan your network and it has signatures for a bunch of known vulnerabilities that would cover IOT devices it's also quite enlightenment to just run nmap run an nmap scan of your entire home network if you have a bunch of devices like this and see how many ports are open because it's really bizarre you have some I find some devices before that have like upwards of a hundred listening ports on

a single purpose device I couldn't figure out what they're doing unless they're just like looping over all the ports to then try a UPnP forward or something like that I don't know but it it's really bizarre

all right well thank you very much if anyone has any questions come find me I'll be in the CTF for most of the rest comm so if you want to talk about Internet of Things I can do that practically the entire concept thank you very much thank you