Extracting Secrets from IoT Devices - Isaiah Davis-Stober

Okay, here we go. So, hello. My name is Isaiah. Uh, I go by Net Code on Discord and Code Nekco everywhere else. Um, I Okay, so this is going to cover IoT devices, mainly firmware. um I have a lot of fun breaking them and doing things that were not intended to them. And also the amount of uh endpoints or servers, let's say that uh these IoT devices talk to that are not wellbuilt, welldesigned, well architected, anything. They were not well built. They were uh the cheapest thing that they could put together. This is really common. But I'm going to go over what is firmware, the IoT, in my opinion, epidemic, uh how to acquire firmware

samples, and dynamic and static analysis of said samples. So I I would say I've been doing computer and electronic stuff since I was instantiated a while ago. Uh in the last couple years, I don't know exactly how long, I've been doing a lot more IoT stuff. It's kind of my one of my favorite things to do. Um, there's some fun things about me like misusing computers, cats, terrible bash scripts, automating things that shouldn't be automated. That's always fun. Lockpicking and bypassing, uh, and tearing apart white white label IoT devices. So, what is IoT? I assume most people here know what it is or like the the general term, but just internet of things. It ring cameras, door like

doorbell cameras, shitty IP security cameras or security cameras. Um, sometimes you'll fe see some really fun stuff like uh temperature sensors, humidity sensors. Those are kind of useful. Um, my personal favorites are when people make things that just aren't needed like IoT toasters. My toaster does not need to be connected to the internet. I if I have to put the toast in, I can press the lever at the same time. This isn't this is not advantageous. This does not help me. Okay. What is firmware? So firmware is the lowest level right we have you know your traditional operating system like you know Windows, Linux whatever but underneath that it can't like this this operating system cannot interact

directly with hardware. It needs something in between and in the case of an like a operating system like Windows or Linux OSX whatever there's a couple layers there's OS there's drivers and then there's firmware. firmware is what's running on the hardware. It's talking directly to the hardware. If you own it, you are God. That is I guess I made two slides for this. That is the main key difference between these. Um there's I guess yeah firmware talks directly to the registry or edit not talks directly to the registers can see everything can do anything. There is no limiting. There's no limitations. Basically, unless the architecture of the chip limits you, you are not limited because of this. There's a

couple major differences. For instance, I'm going to talk about this later, but emulation. You can't really easily emulate something like x8 like a an architecture like ARM or MIPS that's going to be running on your embedded IoT device or embedded device of whatever sort that is not going to uh work quite right on something like your laptop or desktop that's probably running x86 uh whether x8664 it doesn't matter it it's a different architecture it's going to run differently uh different instruction set and there are translation layers uh which I'm going to talk around a bit. But because of that, sometimes dynamic analysis can be a little bit more hard or a little bit more difficult. Um, and because embedded

is like this, uh, there's a bunch of other things you have to deal with. Um, the older libraries, no one updates things either because it's an embedded device with a little tiny chip on it that has the firmware in it and it's typically readon. Sometimes they do update, but it's not as easy as just updating Chrome. Like Chrome is running on top of so many other things. It's talking to the internet. It It's super easy to update. You can update embedded devices, but no one does because it's a pain in the ass and if it breaks, you break the thing. Uh well, I guess I made another slide for this too. So again, why firmware is

so interesting. It's the lowest level. You are God. You can do anything if you have root on a device that is or does not have a traditional operating system on top of it. Well, even if it does, if you have firmware access, you are God. And it's often ignored. People write firmware, it works, then they don't touch it again ever. You see stuff from like 2009, it's like that. You could have updated it a little bit, maybe not giving me root by default. maybe work a little bit for it. Um, and here's why I think it's a fun target. You know, it's low low level is fun. I mean, I hope most of you agree

with that. At least I hope that's why you're here. Um, I don't like fighting AVs. You don't have to fight an AV. What AV is going to You're not wasting resources running an AV on an embedded device. Why would you do that? I don't even know of any. So, no one no one does that. uh all of the security is security by design and no one designs in any security and then no one configures things right. You'll just see Samba just chilling. Just see Telnet. Uh that one's always fun. Uh Drop Bear is pretty common. Just lots of outdated tools that well Drop Bear is not outdated outofdate drop bear. Lots of tools that are not needed are very

out ofd are never going to get updated and they never configure them correctly. they always have embedded credentials or they are like, "Oh, this FTP server, yeah, anyone can read or write." We didn't want to put another account on the device. We didn't want to do account stuff. So, we just said it works and we're good. We're good. We'll ship it. Minimum viable product, boys. We did it. Um, again, minimum viable product. Every single goddamn IoT device is just the minimum viable product we have achieved or we are given a budget of, let's say, 10 grand. Make a thing for 10 grand. That's it's not accurate. Whatever. You make the cheapest, easiest to manufacture thing with the most

garbage firmware that you were able to put together that works most of the time relatively well and then you sell it and there's no security designed in there and you can't really design in security after the fact. That's not how designing it works. Um, and you have a maintenance budget. Where? Can I have it? Please. Please. I want it. Oh, oops. I already have my rant about smart devices. My apologies. Smart TVs. Not smart. Yeah, smart TVs. Can we all agree that smart TVs are dumb? It should be a dumb TV that I put my own smart device on. Go away. I do not want your outofdate Android TV that's just going to join a

botnet. Stop it, please. I should have put the stop. Get some help meme here. Stop it. What? How is it convenient? It doesn't work. It doesn't work. It's slow. It's painful. It's Android TV OS, whatever it's called. It's disgusting. And it's bad. And all it does is give a developers another thing they have to build for that is bad. Just stop. Thank you. Okay, so back to this. There's some I personally I have some major privacy concerns with many of these IoT devices and how common they are. As the talk before me was talking about um like Amazon's I forgot what it's called now, Amazon Sidewalk or whatever. Like that's just a bunch of IoT devices

talking to each other without my consent. Why? Why are they doing this? I don't like this. I don't like this at all. Wish I brought it all to the ground. That's a good idea. Um that's that's one of my big privacy concerns with these. Uh but also these are all talking to some central server that isn't probably well built. Um, and what if somebody gets in there? What could they do? What can they see? All these security cameras that I have like let's say like Arlo or something. You have all these or Google. I think they got bought by Google, didn't they? Anyway, you have all these security cameras. Somebody walks in, gets access to the server that they're all

connecting to, and they can just see all your stuff. Why can they Why am I allowing somebody to see all my stuff? This is ridiculous. Also, another part of this, no one actually sets up u or limits who can view security cameras usually. So, I don't know if you guys have gone gorking on Showdown or Google for security cameras. Um, it no one no one does anything to stop them. They're free. And also as to piggyback off that, why when I'm setting up a security camera for instance, do I need to like, okay, a security camera, I can see why I might need an account, right? I need to or they need to authenticate me against

their server to make sure that uh I only see the videos that I'm supposed to. I only have access to the cameras I'm supposed to have access to. But going back to toasters, why why this is dumb. You have t or not uh fridges for instance. Why does a fridge need an account? Why does a washing machine need an account? What are we doing? Yeah, I do want to sell my customer profile data. That is so profitable for me. I want to do that on the premium device that I bought. Thanks LG and Samsung. Okay, to get back on topic now, firmware. How are we going to get the firmware? We had to analyze it. We have

to get it. So, there are two pretty easy ways to get it for the most part. You can either download it. There's a lot of a lot of places you can download it. Um, or you can get it yourself. I personally prefer the second method because I think it's way more fun. Um, but I'm going to cover a little bit of both of them. Well, a little bit of download and a lot of extraction. I'm sorry. So to download firmware, usually manufacturers will have a copy that you can just download. You have to be careful. Sometimes that's not the entire binary. Like they'll give you a little little tiny uh firmware blob that all it is is the patch, like

the last big patch, which isn't not necessarily useless. You can use it to diff and or to diff it and figure out what major things changed uh what they might have fixed. They don't fix things um or you know they like but sometimes manufacturers do actually just have full firmware blobs just laying around. Often those are encrypted. Um well the manufacturers that do leave full for firmware blobs lying around usually encrypt them and then as part of the update process the old firmware has a key that will unlock the new firmware and then they can continue and occasionally they'll change those keys but often they take a really long time to change those keys. So you can just

keep on using the same key for many many many generations of the same or versions major versions of the same firmware blog. Uh, another common place you can find them is open S3 buckets because all of these IoT devices that they're already out there that are meant to have overtheair updates need to update from somewhere. They need to pull the blob somewhere. And if they don't want to deal with authentication, they just give you an open S3 bucket and say, "Have fun. Just pull from this URI." And you can just take it. Sometimes you need to spoof a user agent, but usually you can just take it. Um, occasionally, well, not quite often actually, you'll find somebody who's

being really nice and just distributing a blob that they extracted. Um, I'm not sure about the legality of that. I imagine that some lawyers might have strong feeling or companies and law firms, whatever, might have strong feelings about, for instance, me distributing all of ECAN's firmware for their security cameras. play. I'm not going to question it too much and I hope you guys don't question it too much either. Um, and then in a similar note to S3 buckets, FTP servers, same thing. They just leave them. There's one. If you want to go download some firmware, it's right there. Um, so when you get into trying to get the firmware, you know, downloading it, you have to worry about encryption, for

instance. Um, if you have or if you are extracting firmware yourself, then there's a couple other things you have to worry about. Mainly bootloader security. If you are trying to extract it over your art, which I'm going to get into a minute in a minute. Um, maybe the bootloader is locked down and that's difficult. Maybe aren't able to do at all. Um, maybe the device has encryption at rest so you can't easily dump the memory. Maybe it's encrypted at runtime. No, no one does that. But maybe it is. Maybe it has TPM or something like that. I believe Amazon uh their newer products, I believe, uses TPM on some of them. Uh which is really rude of

them. I was hoping to look at those. So, what is a bootloader? Uh the bootloader is the first program that's run. It is the very first thing when the device gets power that goes, "Hey, I got power. I got this to turn on signal. We're going to run this little program that's going to build out the file system and grab the actual files from this other area of memory or this other chip. We're going to get the like the operating system. We're going to get the rest of the system that's going to be running. Grab that, put it here, run it. Um, there are some common pretty common boot loaders. Uboot or DOSsuboot, it's very common. Uh, Bearbox, I haven't

really seen a lot of. Red boot, I haven't seen a ton of RT thread. I've actually seen a lot of in really, really cheap Chinese routers. Not sure why they use that over Uboot, which uh, OpenWRT uses, but oh well. So, on to actually extracting the firmware. Um there's a bunch of options. The three main ones would be UART, JTAG or flash extraction um or flash dumping. UART, if you have access to UART, whether like you have depends on how you are set up. It depends on the device, depends on how you are set up. Uh and it depends on if you have credentials. Um, but if you do, you can often just dump every sector like re uh sector by

sector of the flash in order over and over serial very very very very slowly uh to your computer and now you have a complete dump of the firmware. Um I personally don't do this one very much. I don't think it's very fun but it is an option. JTAG is very complicated. I'm going to get into JTAG in a bit. Um, then flash extraction, my personal favorite. There's two different variants. There's on and off chip. Well, chip on and chip off, I should say. Um, and we're going to get to that in a minute as well. So, UART, I don't know if you guys know what UART stands for. I'm guessing some of you do, but universal asynchronous

receiver, transmitter. The trick is RX, TX, and ground. the only thing you're only three things you need to know about it really. There's one other but those are the important ones. Um sometimes you'll see power in my opinion that's a waste unless I have a very specific reason to also connect power. I just don't. I connect UART with RX TX and ground and turn on the turn the device on. Uh important to note the RX stands for receive and the TX stands for transmit and you do not match RX to RX and TX to TX. That does not work. You match RX to TX and TX to RX and ground to ground. That is how you do that. Uh and then the

other thing to know with UAR is baud rate which is basically how fast it's going to be sending information. Uh because it's just a serial connection. How fast is it going to be sending it? so that your device and that device are talking on the same frequency. If you don't pick the right frequency, it's going to look like garbage. Just random bits. Nothing's going to make any sense. You're going to be very confused. Um, this is what UART looks like on a router. As you can see, it's it's pretty simple. That I believe was ground. I don't know. I probed the I forget the order. I probed these to figure out which ones was which. Um, but one of them is power, one

of them is RX, one of them is TX, and one of them is ground. And you just probe them. Like over there, you put it one pin in each and here's another one. This one is actually nicely labeled. They didn't even include power. They got TX, RX, and ground. It's kind of hard to miss. The one over there, um, they didn't label them very well. I had to go hunting a bit. But same idea. They have them there. You They're labeled. You probe each one and as long as you have is it? No. Okay. Well, I for it's not in the next slide, but there's u I forgot the model. It's essentially a UART to USB converter um which allows

you to probe UART and then using a tool like POCOM on your computer, you can just talk directly to it. Uh on to JTAG. I'm not going to really get into JTAG because it is um painful. Uh UART nice and simple serial back and forth super great JTAG is like what if we gave you access to every register of everything and also there's like the uh I can't even use words for this. The way that we're going to make this all work is about as complicated as like HTTP. Why? It's good for debugging. It It is good at that, but in no one actually uses it in IoT devices because why would you use JTAG? That costs money

when you could just use UART and achieve not as good a result but a good enough result for the level of quality we're going for here. But basically, JTAG is much more in-depth and allows you to get a lot more information than UART typically does, but is also significantly more difficult to work with and implement. So, now we have UART on a device. Let's check the boot log. Like, what's going to be in here? You pl you wire it all up. You wire up your art, turn the device on. Let's see what's in there. Okay. Well, we can see it's just trying to get the file like this is the bootloader just trying to get different

regions of memory together and trying to get everything I guess started in the order it's supposed to start in and configured. There's a lot more to this. I omitted most of it because it would have been a lot of slides of this. Um, but this is important. Sometimes you'll just see things like start address. I don't know what's the start address for H. Maybe it's important. Who knows? We can look at it later. Is it the physical ROM or RAM map? Who knows? Maybe that'll be usable or useful. CPU revision, return addresses, where it's jumping to the start of the image. Like, all right. Well, now we know where the image actually starts. Um, what else? Uh, oh, here's another

good one. Kernel command line. This one took me a bit. The usual baud rate I see on uh embedded devices is 11 one was it 115300 or 11152000. Uh and this one was using 38400 which is one I almost never see. Um and so it took me a couple tries cuz I I I just guess baud rate. It's not that many. You start pico com at the highest one and just keep on going down until you get readable output. Um, I believe it's control A and then control D. You just keep on doing that, you'll get it. Um, but here you can see it's the kernel command line console tty s0. Perfect. 38400. Okay, we can do

that. I mean, the reason I got this is cuz I like got that information cuz I already had the right baud rate, but you can see what it's doing. So, now on to flash dumping. So, there's a couple different types of flash. Um, couple different tools and a couple different methods. So, we're going to get into all those different types of flash interfaces. You have SPI, QSPI and I2C. There's also NAND and NOR, but those are not interfaces. Those are the actual types of flash. And for that, though, that in this case is kind of irrelevant. We only really care about how we're interfacing with it. SPI or serial peripheral interface by far the most common that and QSPI which is just

quad. It's like what if we had it four times it goes faster. It's very good. Um but in this case we're going to be mostly looking at little eightpin SPI flash modules which are all going to just be regular SPI um and very very cheap. Um yeah. So to actually dump it we need to use a couple things. We need to be able to connect our computer to the flash chip. We need to get but we need contact on every pen and we need some way of interpreting it and then something on our computer to manage all of that. The tool that we're going to use on the computer is called flash dump and it's

amazing. I very highly recommend for SPI flash. It's uh I believe it's open source and it's I mean it's free. Um, it's awesome and it works really, really well with the CH341A, which is the one pictured. Uh, it's like $14 on Amazon. I cannot recommend it highly enough. I have a couple with me. They're really great. You just want to dump some flash real quick. You plug it in, you run flash dump, it gets you the binary. Um, and it's worked on just about every chip that I've tried dumping. It does not always write it. There's a database that h that it has that tells you if it can or or what it can do with a

particular chip and with most unidentified chips. It can dump but it cannot write. Just don't expect to write every single chip ever. So onto the now the methods of chip on and chip off. Getting back to that. So with chip on uh you are dumping the flash without taking the flash module off the board. It's soldered onto the board. you're not un like not desoldering it. You're not doing anything. You're just attaching leads to the chip and dumping it straight from there because sometimes you don't want to take it off whether you might damage it. Whether like maybe there's fragile components nearby, you don't want to hit them with a heat gun. Um maybe you just don't want to deal

with soldering. That happens sometimes. Maybe you don't have an iron with you or a air gun or whatever. Um all you need for that is an SOP8 clip. Um, and the CH341 and Flash dump. Those three get you everything. Like in this case, this is a Sen, how do you say their name? It's S E N G L E D. I don't know how you say their name, but they do a bunch of IoT garbage. Um, and in this case, the chip is just chilling on the board. There's nothing really in the way. Sometimes you'll find little components that are right up next to the IEC, like the integrated circuit, I should say. Uh the little flash

module. Usually those are for power to make sure that the power going to the chip is as clean as possible, which is kind of important. Um but as long as there's nothing too close, you should be able to just clip on. Um here's a couple other the ones that I was able to just clip on to without too much hassle. Um, and then for once you clip it on, you just run flash up. I have the command somewhere. I don't know if it's in the slides, but it's a pretty simple command. Um, and that will get you well, essentially what it's doing is just sending the signal to the flash. Hey, I want what you have right now.

Please give it to me. And then the flash just gives it to you and you just store it. So for chip off flash dumping, it's usually reserved for slightly more complicated things, but sometimes there's a chip right next to that IC, like right next to the flash module on the board, and you just can't get a clip to stay, and so you just desolder it, and you know, it happens. Um, but often if I'm doing chip off, it's because it's a type of flash that I like the C CH341A just can't really do or I don't want to use it. I'd rather use an X gecko, which I'll get into in a moment. Um, but for instance, here there's a

little I or there's a little tiny chip right here that means I can't get the clip to stay at all. It does not fit. Um, so I took it off. Still fits in the clip, but now it's not on the board. Um, and now you can use flash dump, which I believe I already explained this, but it just or essentially sends the signal, hey, I need all of the memory. Just give me all the memory you got, and it just gives all the memory it's got. It's great. It also powers powers it. The most of these programmers will power the chip, so you don't actually need power to the board. There can be complications with that uh

because you are putting power on the bus. Um some chips are weird or some boards are weird, poorly designed I should say and they will uh act weirdly if you give power to the bus that uh the chip is on. So the Xeeku is basically a much larger uh more useful I should say chip or flash extraction tool. Let's say it does uh there's a couple variants, but pretty much all of them allow you to use m or allow you to dump much larger flash chips. In the examples I've been using, it's all eight pin SOP8 chips. But let's say you want to do a 48 pin chip or a 56 pin chip. I don't know,

maybe 24, like some really much larger chip. Usually what you'd see on higherend security cameras like a Hike Vision for instance, those typically have much larger ones. Uh sometimes nicer routers will have that. Um, and those can get much more complicated. Getting like solid stable contact on eight pins is a lot easier than it is on 56. Um, and yeah, here's the main use. This is on a Hike Vision security camera. Um, there's have a big piece of Toshiba flash there at the bottom. And you know, if I'm or I'm planning on dumping this very soon. Um, but when I do, I'm going to be taking that off because I'm not putting a clip on that.

That doesn't work. the pens are too close together and it's too small. I'm not doing that. Okay, now on to the analysis, the fun part. So, static verse dynamic analysis and then common common embedded file systems. Common in embedded systems, file systems, I'm sorry. Um, so first let's check if the dump is encrypted. So the easy ways to do that usually are check the entropy with binwalk binwalk capital E or hyphen capital E. Um and then check what different file parts of the files look like with bin viz which is an amazing tool biniz.io I should say. I believe there's another tool called bin viz which has no relation and is a couple years older like from 2009 that's a guey

tool that you install that it it's pretty good but I don't know I just use bin viz.io. It's better uh for most things. And here you can see, hey, this all looks kind of random. This is very high entropy though. This is a little interesting, but oh, okay, here this is clearly structured data. We can kind of see, okay, this is text. Not exactly sure what that is. And then the white is zero, but clearly there's something here. And Ben Walk would agree. You know, a lot of it has really high entropy, but there's clearly some other stuff in there as well. It's good. This is not good. The there's no there's it's all one. There's the entropy is so high.

It is no there's no way it's not encrypted. And yeah, this is encrypted. Um, so now we want to actually analyze it. Let's say now we we have a decrypted binary. I'm not going to get into actually going to the decryption process. There's some great blog posts on that and I would not be able to do them justice. Um, the easy ways to do basic static analysis are going to be stuff like binwalk, strings, gedra, and xxd. xxd is amazing. Um, and then I'm also going to go over firmware the fact the firmware analys firmware analysis and comparison toolkit which is a fun thing that you can self-host uh that does a lot of this for you. Um, but

binwalk well never mind. pinwalk basically goes through. Wait, no, I have a slide. Never mind. Sorry. Sorry. Um, so now we want to determine what we're working with, right? So, we maybe should have determined this earlier, but let's say what's the CPU architecture? Realistically, it's probably Mips or ARM. A lot of it's MIP. Um, what firmware or what might the firmware be based on? Like, if I'm tearing apart anything from ECAN, I already know what firmware they use. they just modify it slightly depending on what they're doing. So, I'm just going to be like, "Oh, okay. Well, the root password or the root credentials are going to be root TINA. That's it's not I don't need

to tear it apart. I know what the root password is already." Um, but you might want the kernel version, for instance. See if there's anything that would allow you to escalate privileges, which you never need to do. You're always root anyway. You're either completely unprivileged or you're root. um because every they just run everything as including file uh web servers and then if there's busy box which I'm going to get into a moment what version is it how out ofd is it okay so now we're going to continue with the extraction I'm sorry I'm jumping around a bit um let's say we have this blobbin which is from this uh smart home hub like I don't know what to call. It's

a smart home hub. It's terrible. Um let's say I want to know what's in this. I could manually go through and try and find things that look like they might be part of uh the file system or go through the boot log and say, "Oh, well, starting addresses here. We'll just keep we'll take like I'll extract that part to the next place that says it's starting something." Um, but Benwok just goes through and tries to find magic bytes for files and file systems and then helpfully extracts all of it. Um, importantly, you need I think I have a slide about a couple other tools that needs but or dependencies, but um, this allows you to get a pretty darn good

idea, if not everything of from a binary file to, oh, we have a file system. We know how these are put together. uh we know what's in here and we know a bit about it like if it's little ending or big ending whatever. So on to file systems. Common file systems that you're going to see are going to be Squash FS and JFS2. I the journaling flash file system 2 one was terrible we had it was so good we had to make second. Um so the very key difference here is Squash FS is read only. You cannot write to it. JFS 2 is writable. Uh so usually you'll see JFFS2 for uh user configurations like oh you want to set

the Wi-Fi password like uh you want to set the SSID and the uh password for your Wi-Fi network that this is connecting to that will be stored on the device in the JFS2 partition or volume forget the term they use. Um but to extract them we're we need a tool to extract them. In this case, we're going to be using this is okay, never mind. Uh, at some point, oh, yeah, here it is. So, to extract them, we're going to be using Unsquash FS, which works pretty well, but has some problems because vendors hack together their own implementations of these file systems because they're [ __ ] Um, and Sasquatch is a patch that for on Squash FS that handles most

of that very well. Uh, and then for JFS2, we're going to use the tool called Jefferson, which um, if you have these tools installed, Binwok will just run them for you and just continue extracting everything. It's very helpful. So now we have binwalk or we've run binwalk on this and we have a decent idea of we have the files. Now what's the layout? Personally, I'm a big fan of Exa or XZA and tree. So, here is what or part of what um forgot which camera this was. Um oh, this is an ECAN camera. This is what part of their directory layout looks like. And you can see there's some interesting stuff there. Like for instance, pseudo in it. What's pseudo in

it? Is that the next? No. Okay. Next up or next. You want to know, let's say, how are these file systems laid out? How are they mounted? How are the directories laid out? Because if you want to actually get something here, if you find something but it isn't being run, then did you really find anything meaningful? Like if it isn't actually running on startup, is it useful? An easy way to find like find out what order things are being run in, if they're even being run at all, how they're mounted, whatever, is fab or fs tab, uh, init tab, and then whatever other init script they're using, like pseudo init. Um, and if you want to find

particular files, I assume most of you guys know GP, but or find, but rip gp is my personal favorite. It's just recursive GP. Um, and then just Exa or some other way of just looking through the file system tree. Um, and then also guessing I mean it's configuration files where there'll probably be been a folder named configurations or config or something like that. Just look around. You'll probably find them. Um, other useful or other tool that you want to look for is BusyBox. Um, it's a small utility for embedded Linux. Really great. It's a single utility, does so many things. It only does the things you compile it for, but that can be whatever you want.

Basically, it's called the Swiss Army. I think their slogan is the Swiss Army knife of embedded Linux, which it is. Um, to use BYBox, as long as you have it compiled to do the thing, you just sim link like, oh, I want, let's say, I want O. Did I compile Bizzy Box with Bizzy Box with O? Perfect. Then I just make a sim link in user bin called O. And now I have O. It's great. Um, and if you want to find out what version it is, it's busy box help. This is what running on my laptop, but that yeah. So you're looking for configuration files used GP rip gre find whatever you found some. Let's say like

samba config or samba password. It's surely slashing in case you want to access. This is for the smart home pub. I don't know why they use sampa, but you can just you can just walk in. Um, oh, usually they're going to be.com or I ini. So, you can usually just find and grap for inis and conf uh files. Sometimes they're going to be embedded in uh like custom applications or custom utilities and you're going to need to like run strings on those or PJ or something. But often you can find lots of fun stuff from conf and IMI files. So now you want to find the misconfigurations. It helps if you're somewhat familiar with how the Linux

system is or like how this system might be set up on another more normal system like how is Samba normally configured? How would I normally secure Samba? Why is this not secured or how is this incredibly unsecured in some way? If it helps if you are somewhat aware but usually you can just read the config files. They're usually documented, make lots of comments, or you can just guess like, oh, we declare a URL here, then we declare a username, then we declare a password. I wonder what it could be. Maybe maybe these are for the same thing. And not every misconfiguration is exploitable. As I mentioned earlier, and I'll mention it again in the future,

sometimes something like Tnet, maybe it looks like it's being run, but it never actually runs. Like it's installed, the configuration files there. Maybe the script even references it, but it just doesn't run. Like that partition isn't mounted. So now, let's say you want to find a hard-coded CRED. Like, let's actually find some secrets here. Password file, shadow file, configuration files. They're usually plain text. Um, and a couple custom applications and scripts. Let's run strings. See what's there. you can usually find a lot of root passwords or various other passwords that or I should say credentials because you can also find keys. Um you want to locate custom utilities and scripts. I mean a basic understanding of what's what you would

expect to see in Linux helps. Like I'm not going to say oh my god O I wonder like I wonder what custom utility this is. Like I know a is there by default like it that not by default but I know a is a Linux utility. I'm not going to be looking through to see what's up there. Um, so if you know a bit about Linux and common utilities, that'll help not waste time. But let's say this custom script pseudo in it. I mean, I'm just going to read through it. It's like 400 lines. I think it's not that long. It's just bash script that runs on startup. What does it do? Well, it makes the JFS2 partition

and what else? It makes a UBI or UB IFS file system. Um, it tries to update if it can from their very bad update server, but it tries. Um, and it just, you know, starts all the services that need to start. Um, and I'll get back to that pseudo in a script in a bit. Uh but you know common things you'll find in custom scripts are embedded credentials you know possible endpoints is it reaching out to a particular server for something overtheair updates like the process to actually update is often going to be initiated in pseudo init because or like the init script when it starts because it wants to say hey I just started I'm

running this version of firmware is there anything newer can I can I get anything better than this um you'll often find tnet drop bear atrocious error handling. Error handling is not well done by most people in bash. Um you'll actually understand how the file systems are actually like are set up which can be really important for this. Uh and then you'll sometimes try and find it true enlightenment but that hasn't happened for me yet. Um embedded credentials are sometimes hashed I should say often should have been sometimes. Um, as you can see here, there's it. It's like it you can cap shadow. You can also just cap password, get another account, cuz why not? Um, I don't know

how many of you guys crack passwords, but just for most of these, just Google the hash. Did somebody already crack it and put it on Git or GitHub? That's really nice of them. You should do the same and be a nice person. Um, but you know, it's handy. You don't actually have to you don't actually have to try sometimes. Like there there's the password. Easy. Um, because they're, like I said, often not encrypted. And especially in configuration files, no one no one actually hashes passwords in configuration files. Um, overtheair updates. That can be a pretty big security flaw sometimes if the server is not secured and they they sometimes are but not always and often not well.

Um but like looking into the update process you can sometimes get some really in valuable information about how it's updating what's running why. true enlightenment. Uh, Telnet, it's Tnet. It's Tnet D. It's very simple. It's usually built into busy box. Um, yeah, it's it's usually there without it's usually configured, not always run. Here's pseudo init again. Here's the same scripts pseudo init from two different file systems. And you notice on the same device and you notice one of them Tnet's not initiated or and the other one it is. I don't know why that is but that's not necessarily a vulnerability. If that one says, "Yeah, we're initiating TNET or starting Tnet." But this one is the

one that actually runs and isn't initiating or starting TNET, it's not really a vulnerability. Then uh usually for SSH, you'll see drop bear and it's often incredibly out of date, like 5 to 10 years out of date because reasons no one wants to update. If it works, why would you break it? um appalling script handle like error handling in scripts or errors or not or custom applications. It's it's always bash. No one no one handles errors right in bash. Uh there's some fun, weird, and wacky file system setups as I mentioned earlier with JFFS 2 and how it is layered on or like overlaid on top of uh Squash FS because Squash FS is read

only. And so if you want your uh main file system to be read only, you cannot write to it at all. Well, that's good from a security standpoint or a like reliability standpoint. like a worked update is not going to destroy it, but you still want, you know, to store user data. Well, now you need JFS, JFS2, and you're just going to essentially overlay it, and it's I don't really know how to describe it better than that. It's a little bit weird. It's fun though. Um, and it allows you to store user data. Uh, now looking at custom utilities, like one I think I'm going to mention is CI apps. um just strings, Gedra and XXD, it's basically all you

need. You really just need strings for the most part, but Gedra sometimes is really really helpful or whatever your decomp decompiler of choice is or reversing tool. Um I think binary ninja is pretty decent at this, but I think Gedra has more plugins to support more strange architectures. Um here's some fun stuff I found in some custom utilities. off. The key is a lot of ones. That's great key. Uh there's CI config. See all those servers there. Wonder what those servers are for. Spoiler alert, it's for uh management and the or how it's streaming video out. But you can usually find a better credentials APIs um sometimes keys as well. uh the API endpoints, IPs and ports,

host names or domain names, whatever. Um and URIs for like update servers for instance. Um the firmware analysis and comparison toolkit pretty great if you have a lot of firmware or you don't want to go through every single one of these steps in depth on every single file in a in a dump. It's It's pretty great at that. Um I'm going to see if I can do a demo of it at some point, but um we'll see how that goes. Okay, so dynamic analysis. If you if you want to run this, I believe I mentioned earlier with emulation, it's not super simple. Uh QMU has uh a great a couple great plugins for MIPS and ARM that allow you to or

extensions for MIPS and ARM that allow you to get most of what you want. Um sometimes all of it, but my preferred way to do it for the most part is just over UART or JTAG. Just try and get a UART shell and see what's up. Like what what do those file systems look like that? like the way I figured they look like I was pretty sure they work like this, but how do they actually work in practice on the device? Are permissions what I thought they were? Are these credentials being used where I thought they were? What does this application do? Like if I run it, does it do anything? I can look at network

traffic and see if it does anything. But or maybe I can look at maybe it'll give me good output on UART. What does it do? I mean, for the UART shell, you need RXTX ground, guess the baud rate, and then hopefully use the credentials you found in the firmware that you were hopefully able to get. Um, and now you have a shell. Like, what what's there? Shell options. I mean, JTAG has some ways of getting shells, but it doesn't really use shells as much. SSH sometimes, like if SSH is enabled and you already got the account credentials for SSH or have SSH access, just use that. Tnet, same thing. And then network traffic. Well, I

should say I'd like to do more about dynamic analysis, but um it is significantly more in-depth than static analysis and I don't have enough time to I don't think to do that. Yeah, I don't. I'm sorry. Um however, there's some great fuzzing tools like AFL AFL++. Um, one of my friends actually just got um he finished his rig for AFL++ to uh fuzz tapo cameras. So that's what fun project. Um so network traffic that's a whole another section of this. These are all IoT devices internet of things. They're connecting to servers. What are they doing? How do we get it? What's in it? I mean for the network setup I mean the way I would I wanted to do a lot more

with this uh but I was moving and I had to rebuild my network and I did not get that done in time. I apologize. Um but usually you know dedicated VLAN man everything you win. It's it's a victory. Um I mean analyzing network traffic you're just doing a TCP dump and then wire sharkark. It's not terribly diff or complicated but you know it's fun. Um, one fun thing is lots of, not lots, some of these use SSL. They never ever ever use certificate pinning. They just don't bother. You can do whatever you want. Man in the middle wise. Um, and essentially the same deal. It is essentially the same deal when it comes to whatever apps they use to talk to

these cameras. Um, and you can often find credentials in those. Um, I was hoping to look at um fact here, but we'll see if I have time. Um, here is here's fact. Well, wait hang on. Yay. Here's here's fact. So, let's say I want to look at this ECAN doorbell camera. I uploaded it to this, had it go through a bunch of analysis on this as Oh god. Never mind. I don't have internet here. I apologize that's not happening. I'm

sorry. Okay. Sorry about that. I was hoping to do this. Um, so I guess in conclusion, ways the firmware can firmware can be better secured. you know, locking down the bootloader, making it difficult to dump over UART, not just divulging all the information ever. Um, stop shipping things in debug mode. If you don't ship them in debug mode, it makes this a lot harder. Stop embedding credentials. Just stop stop dynamically generate them. Stop embedding them. Um, and encryption at rest. Also, you if you really want this to be well done, use TPM or some other way of holding on to keys of some sort, but that costs money and as a result, it's never going to happen.

Let's be honest. Uh, possible impacts of some of these things. I get to have fun. Hope you guys get to have fun. It can be an initial foothold on a network, especially with routers and IP cameras. Um especially routers, botn nets. Who likes IoT botn nets? Mariah is everywhere, but there's so many different versions of it. I don't even know what to call Mariah anymore, but you know, Mariah, Mazy, stuff like that. Uh, does anyone have any questions? Yeah. If you have an X Gekku, is it worth getting the CH438? I have both, but I I don't know. The CH 341 is it's really convenient. Like if I want to use an X Gecko, I have

to open their software which doesn't like Linux. Um there's an open source rewrite of it that is much better and works on Linux, but it's a lot more effort than running Flash Dump. And for $14, I just have both. when you're uh when you're trying to flash off the chip, how are you doing chip off? Like if I'm doing chip off extraction, uh for chip off, actually I should have had a picture of one of the clips here. Um I'm basically taking I'm desoldering the chip and then there's a little holder that it will basically pop into. So as long as I clean up all the solder off of it, so it's just the chip and its pins, there's

nothing else. I can just set it in there. it'll clip in and then it makes pretty pretty good contact with all the pins. Worst case, I you'll have to sometimes solder like if you have a really weird pin or mangled pin or something, sometimes you'll have to resolder it to a daughter board or like a sacrificial board and then plug that in. But usually you don't need to do that with some of the chips that are running dual uh architecture such as ARM 32 and now res 32. Um are you seeing the tools being updated for the access like the like uh QMU? Yes, I believe there's some pretty new version or extensions for QMU

that allow emulation of those, but uh I haven't encountered those in IoT device or the cheap IoT devices I end up getting recently. What was the uh tool that you were using to uh attach or like connect to the like connect to the chip? Yeah, the specifically uh to the pads without having to solder to them. Oh, okay. like the what tool I was using for test pads. Uh that is a PC bite. Um that cost I think like $180 uh for the whole kit and unfortunately it's worth every penny. It's amazing and it gets rid of all the pain of trying to make contact with a test point that doesn't have a solder mask and I can't solder to it for

whatever reason. And it it it's so beautiful. The people who put 7 millimeter test pads need to die. I'm sorry. Hot take. They need to

[Music] die. How many fros? Has anything surprised you like how well something is? Um

really the only one that's like, oh, this is kind of disappointing is like that's slightly different, but like whenever I get a cheap router and it has OpenWRT on it, I'm like, well, that's not very fun. I wanted something terrible. Uh, but I don't know, Amazon has pretty good like they have they have a whole team dedicated to theirs and they don't even label the test pads. They label them like TP 0 through 50 and all right could have just labeled them something else but whatever. Um,

if you just want to get started, I would suggest going on Amazon. I should have included a picture of this. If you go on Amazon and search doorbell security cameras or doorbell cameras, um, look for one that's kind of like rounded, kind of thick, doesn't look very expensive, and it's about 20 bucks usually. Uh, there's like eight variants of them. They're all made by ECAN, and they're all garbage. It's super easy to get uh like super easy to get UART on. They're easy to disassemble, easy to get UART, and the flash chips are not hard to get to, and their security is abysmal. There any other questions? Repeat which part? Uh, ECAN or EKN, I believe, or an

I think EN. Uh they make very very cheap security cameras, doorbell security cameras, uh light bulb security cameras with pantil zoom that are running the exact same firmware as the doorbell ones but with pant tilt zoom. Um if you go on Amazon and just search doorbell security cameras or doorbell cameras, they're all over. And if anyone wants to ask me about those later, I can I can show you some and show you pictures and I'll show you on flash with secure spine or flash not on IoT devices. They they don't that costs money. No one no one wants to spend money on this. All right, I feel like my time's probably up and I don't think I get

there. Are there any more questions? Is that it? Awesome. Thank you.