How to sink a UBoot: Understanding bootloader attack surface - Keith Makan | BSides Cape Town 2023

does this work no there we go it's on okay has everyone settled in sweet okay so uh my name is Keith McAn I should try not to lick away from the mic my name is Keith McAn um I'm from km sick I'm one of the Soul responses this year yeah um so yeah let's get started to today's talk is about how to basically how to understand bootloader attack surface and um um this talk basically came together at as uh as a sort of conglomerate of a lot of like bootloader reviews I did over the last two two years as part of the contract work that we do um so I'm basically going to talk about bootloaders in a r

generic way talk about some of the architectural stuff that you might expect in a bootloader talk about how multi-stage bootloaders work and basically if you're developing um something with UB boot you're looking at a youit instance what what would you want to see that could go wrong and uh what's like a red flag in terms of a of a configuration and then I think um something I rarely do is actually talk uh a bit more extensively about how to fix it so I've got like a little Grid at the end of the talk with six points um some of the more important things I think you've got to pin down to make sure ubit instance doesn't uh give your

attackers too much Freedom um so yeah let's get into it uh oh this is not the way I expected the slides to go anyway we'll roll with it um yeah so that's me um I like uh hiking fishing and running and that's my blog and uh I started my own security consultancy trying to help out folks where I can and um yeah I wrote two books um and that's what I look like when I'm hiking and that's what I look like when I don't hike so yeah um I don't want to get too much into this slide it's always the most boring one so yeah um agenda what a bootload is um what's you boot what does

a multi-stage boot process look like sort of generically um and then we'll look at some vulnerability analysis of uboot what where do where do UB boot's bugs typically come from um and uh how many have there been like I've got a nice overview of the reported bugs for uboot um so yeah and then a methodology for attacking it and how to secure you boot so bootload is 101 um why do you even need one well in um typical embedded setups uh most of the ones you'll see today there's something called a system chip and there's local resources inside the system on chip besides the CPU so you'll have caches um static Rams this is what SRAM stand for and you have

eusers and a bunch of other things um and these are accessible to the stuff that runs first but most of the OS is not going to be um accessible in the initial State um because the resources are too limited for instance a arm cortex M4 only has .5 gigs of uh of SRAM you're not going to fit an entire Linux instance in there maybe there's some variance of it you can but it's not rich enough to actually um give you everything you want so there's got to be a bridging process from these isolated local resources to the stuff that's stored in your USB your SD card and your serial flashes and um to some points you

you typically have your ROM burned in by the vendor so there's not much dynamism to it it does uh very few things typically it will I'll get into this a bit more but typically it will do some initial setup it will uh check that uh the local resources are accessible and sometimes in ROM code you maybe want to make us as the euses and TPMS and uh you also want to host something that's Dynamic enough that you can boot multiple os's off of the other storages um so yeah uh quick quick overview of bootloader types this is probably very simplified but there multi-stage and single stage single stage of like archaic things um they were usually just

the ROM boots up loads the OS into memory and then goes for it um and that usually meant that um Ram was pretty simple to set up um you only had one kind of ram there probably wasn't an mmu but nowadays things are more complex you have multi-stage boot loaders there's multiple parties um involved in developing an embedded device you might have one party that develops your um first stage bootloader you might have a vendor that burns your ROM and some completely other team another team that bus your OS they've all got to work together the only way that works nowadays is to have a multi-stage boot loader and then there's also um things like verified Boot and trusted Computing

that compfy matters a little bit verified boot is what boot uh uboot call secure boot there's other variants of this in in other boot loader technology um and basically verified W allows you to cryptographically sign and verify um what you're actually loading on on your board um and it prevents obviously all the all the great things that symmetric cryptography prevents um gives you nonre non-repudiation everyone loves pronouncing that word non-repudiation make sure um you you actually have to be authenticated to change the OS and then obviously there's trusted Computing modules that uh allow you offload crypto and isolate computation to to more controllable contexts um and just so we have sort of the mind frame uh correct

about how what kind of goals we have for bootloaders I break it down to four sort of basic things there's a lot more that can go in this slide but you want to Bo a verified image you want to make sure your trusted Computing is set up properly there's no holes you can't like poke the TPM make it do stuff it's not supposed to or leak bits from it you want to prevent anti-rollback um one of the things I think in the reviews I've seen is um typically people will secure a whole boot loader and not even Implement anti-rollback so that means you can just flash an older version of ubot with a whole bunch of bugs in and

it doesn't really you not really gain much ground in in security um bootload of security land um and then obviously the main thing is initialize the kernel securely because at some point it's got to hand over um uh the it's got to hand over to the kernel and have every have the um embedded host set up so that the kernel has access to the resources it can set up its user space the mmu is running properly and isolates um address spaces properly and all the sort of security Rings uh um and isolated contexts in the board are are working properly there's no way for the kernel to use the bootloader or the bootloader to abuse the kernel um so specifically

what we're talking about here is uh I mean there's lots of bootloaders we could talk about um but you would I think is the most accessible one it currently runs on something like 94% of all the um inited devices you'll see um and that's because it's just accessible I think it's one of the oldest um products and supports a wide number of of architectures there are growing competitors um to that are sort of coming up as well but I think ot's the one Everyone likes and it's the easiest to build as far as I know so it started out as something called PPC boot um which was a power PC um uh bootloader very small only supported one

architecture and this guy named wolf gong dank um forked it and renamed it um mostly because Sorge Forge doesn't support um numbers as names so it used to called 8xx wrom and he needed to rename it so he just called it PPC Boot and oh wolf G I think from as far as I know he passed away last year October so um that's a bit of sad news I didn't mean to say it in a sad way but just just as an info point but yeah amazing work um obviously must have been an amazing guy um and yeah he released it on sge Forge in July 192 uh 19 July 2000 two years later they added arm support

So PBC boots technically the wrong name for it at that point um cuz it boots arm as well and then um a month basically a month later they added a bunch of other support and by 2004 it was supporting like over 200 architectures and board types so grew pretty quickly um and a lot of hands there were a lot of hands in in making it so accessible easy to use um so okay let's talk about uh sort of the normal boot process um usually what happens is you start off with the ROM boot it's know as the first thing to execute and um what happen Happ s here is again there's not much to um access

the resources are limited and what needs to happen is something needs to get uh the second stage boot loader the secondary program second program loader or secondary program loader into um Ram that actually fits it and that's what the ROM that's the main job of the ROM turns on finds the SPL and loads it into um um appropriate Ram additionally it could also verify the SP SPL so it has um some of these oh I have another slide where this totally defeats the the animation anyway um so so what happens is you have something like a Serial flash that maybe holds yourl it gets loaded into SRAM by the ROM code and uh what you could have is uh in your e

users you might have a public key um that's used to sign the SPL and and this is how the verification work it pulls the public key out and and verifies it and make sure it can run um you don't uh I don't know if anyone actually uses uh symmetric cryptography like manually but you don't typically encrypt the whole thing you just make a digest of that and you sign hash and that's a lot faster um so yeah that's typically what the ROM does second stage is the secondary program loader and um this I think uh basically spins up a lot more of the more complex um peripherals touch screens um gpio stuff you probably want

to do this year because um there you don't have a lot of power in the in the in the um in the first boot process so it's been verified by the ROM and its job is to actually um load the boot loader that loads the r um and uh that t you can build you can build ubit as a SPL or TPL and all it does is it splits out another binary and you just package it so that that it can be found by the board board your um your uh your booting um and what I've seen also people like to do anti-rollback checks in the SPL so it looks at the euses make sure that

it's burned and checked properly if you boot a new one Burns a new fuse so you you can't roll back uh to to the previous version um if you're reviewing boot loader code I suggest you actually check this because it's very easy to skip um anterior back checks and yeah that's that's typically the sp's job spin up um more peripherals make sure they're working verify the TPL and brunch Branch to it and the DPL pretty straightforward bootloader from from sort of the kernel if you work with kernels a lot this basically the thing that get you going um and yeah it job is pretty simple just load the kernel make sure it gets the command line make sure

that um the the mmus are actually uh spun up and working properly oh the other thing I forgot to mention is uh external resources have work on clock rates and there might be multiple clocks on the board so another thing the ROM does is make sure that the clocks are initialized and that they they're actually taking and we're aware of all the sort of frequencies that we need we need to be operating at okay so that's pretty straightforward boot process um I probably said a whole bunch of things wrong but you can uh you guys can talk to me about it at the end um so where do boot where do you boot balls come from

um and from what I've learned uh is is typically three main sources a lot of people talk about boot loaders and um they I mean they're exciting I think the sexy things to talk about is Fault injection attacks or you know these very Advanced um very Advanced uh electromagnetic like glitching they're very cool attacks um but I think uh they require a lot of access and typically they work on very small Windows um that the bootloader op um so to me they're like the tip of the ice workg there's so much other stuff that can go wrong from the security perspective you could Implement your crypto wrong you could uh have race conditions in how you're

reading your SD card there's there's a plethora of issues Beyond just uh skipping a voltage or clock cycle and um skipping of instruction um the other the other source of uh VES as typically with open source um um embedded stuff is drivers you've got uh your your um the the people that make your sock are not the people that are making your Wi-Fi CHP they're probably not the people making your flash driver your mmu and there's got to be drivers and software that actually runs this stuff and typically what happens is it all slaps together pretty fast sometimes you get a reference driver in your buold which has no security bolt intern it just meant as

an example and people miss this so one of the big sources of bugs is third party drivers sometimes they don't even check like you'll see bugs where people load um firmware into a an LCD screen or like a touch screen and don't even verify the touchcreen um foam Weare because there's so much else going on everyone's worried about anti checks and uh uh um aslr being implemented there's a bunch of stuff that that goes on in third party drivers that opens up a lot of bugs and then notoriously for uboot is um its configuration is driven by setting environment variables um so uh typically what happens is you don't set the right environment variables for what

you're actually um rolling the board out for so um there's a bunch of Environ variables there's a bunch of stuff you need to implement um if you're if you're doing TPM stuff um a lot of TPM bolts don't come with their own random number generators sometimes you need to implement them yourself um and you know this type of stuff can go wrong so I took a sample I'm going to say it's a sample because um you know there might be someone reporting a bug in uboot right now um but uh I took a sample of about 35 37 bugs um in uboot I think for my effort that's all the bugs I could find um but yeah anyway this is what

they look like so on the on this a you'll see a bunch of bubbles the bubbles Loosely reflect how many bugs were reported in that year for that version and the version is the maximum affected version so a lot of this data if you see something affecting the 2009 version it probably affects the 2018 version too so as the as the um it rolls forward um 2014 version 2010 gets more and more uh vulnerable I guess so 2014 2017 um not not many bugs reported basically just as implementation was using a static CBC a static IV in CBC mode and there was foot passing bugs and one thing I think you'll see with uboot is uh because there's a lot of moving

parts and it basically passes files and as you know um open source file passes are notoriously like swiss cheese they have tons of bugs in them it's very difficult to pass pass a file sounds simple to find some offsets in a file but can go horribly wrong you can even make a touring complete file pass if you don't what are you doing and that's what happens here so people I think um if you look at for instance uh 2019 it's a huge amount of bugs reported about 18 um and uh all of them were NFS Most of them were Network file system bugs and network file system bugs are basically file Coster bugs too so what it looks

like someone just made a NFS fuzzer and aimed it at everything that passes uh that file system and they found a bunch of U boot bugs so really cool stuff nice thing about net file system stuff is uh they might be used to boot files over the network so if you intercept um the completely unencrypted tftp um traffic that ubit uses you can pop a shell um or probably make ubit do something it's not designed to do um the other thing is foot image passing so a fot image is something that holds the device tree um for uboot it tells you which components are loaded it has the it tells you like where the um public keys are what's

being used to sign it and um it's something that you pass you would to tell it where all the non-discoverable components on the board are um and it needs to pass this image and it turns out there's a bunch of bugs in uh in the way that it does that too and then um the other scary stuff is in 20 2022 uh there was a bunch of memory corruption bugs everywhere uh well we only reported people only reported seven and they also have to deal with file passing adjacent stuff so the squash FS was was pretty vulnerable um anyway I'm tired of talking this graph I think the next one's pretty cool as well um so I

clustered them according to um affected area too um so it's nice to look at like what really causes all the problems and um about half the bugs ever reported you boot was from the networking stack and uh the typical so at the bottom bottom half of the bug bottom half of the block you'll see the cwe identifier which is telling you what kind of of bug was reported and typical memory corruption stuff because you is written in C so you're into a lot of these issues dangling pointers double freeze um read reads out bounds and copies without size checks um which are pretty easy to identify with source code review so it tells you that not a lot of money going

into securing you boot unless it's like your own build um a lot of the public stuff probably doesn't get enough attention and um I think if you're used to reading a lot of uh bug reports um there's a bit of asymmetry like on the one side you have a bug in the USB driver which is a heap overflow on the other side you have tons of bugs in um NFS stack and the UDP you typically you'd see U USB as part of the chunk of bugs because it's easier to fuzz for there's tons of USB fuzzes so to me it looks like there's probably a bit more USB bugs in uboot um cuz 2% doesn't make

sense if you have like 50% um uh with in in bugs and the rest of it I think pretty much handled the file system handling bugs make sense uh if if you fuzz the the NFS stack as well um so okay yub's attacking ubid as a methodology so there's four typical steps you can jump in at either one um depending on how you're heading the board but uh typically you'd want to enter up boot and um there's a bunch of interfaces on the board that you could traditionally use so one of the most most common ones is a uart console um they put these on embedded boards because uh when you roll out when you're

building an embeded board uh there's no screens there's no way to check that it's working properly um I heard this story about this very interesting job someone had um at uh at a when you make bombs like very dangerous uh weapons um someone needs to test fire these bombs so and they've got embedded boards on them and so they're receiving signals they're telling you when to trigger they tracked the bomb state it blow up is it going to blow up should I stop blowing up um that's an embedded device and so if you if you fire the bomb it doesn't go off what do you do do you go home no someone has to go out pick up the bomb

drive it away and then explode that bomb somewh safely so that's someone's job they have to go and pick up a bomb that didn't explode um and tells you a story about how difficult it is to engineer and bed a devices sometimes you can't tell what's going wrong unless you leave some kind of way to to listen to the device and and uart console is everyone's favorite way to do that it's a serial connection I'll show you what one looks like um if you're going to get into um de compiling um inter Riv engineering beded boards finding your consoles and hooking up uh rs232 to them and seeing if you have a Uhl it's one of

the most fun things to do um I'll talk a little bit how to do that other interfaces are things like J tag you could um you could also pop off some serial flashes listen to what the flash is doing um and we'll see how to glitch there's a simple way to glitch a Serial flash or um uh some some ra RAM chips uh in a second very interesting technique and then there's more advanced stuff like glitching you can do fault injection or or or pin toone once you interrupted boot which you made sure that the boot process doesn't have what it's expecting um typically because of the way U boot is designed uh you might

be able to drop a shell so there's this thing called Auto boot in in uboot and it basically um almost like like a panic design like if anything goes wrong drop the shell and then it gives you access to a bunch of uh I'm sure you've all done that like I I grew up uh I'm one of those people that always on campus would deal boot Windows and Linux because I didn't want to I didn't want to be with the windows crowd cuz they're pretty basic and I didn't want to be completely in the L shot cuz they're weird so I would I would I would deal booted and uh when you do that you wake up every

morning with your you know your hot in your throat and you go am I going to get the grab prompt or not today like so so having a having a a shell in your bootloader is is really good because it's a it's a it's called a fail open State you get access to all these commands and you can manually um you know piece together your boot process and and fire it off once you've got a shell you want to actually modify the boot environment to do interesting things and the interesting thing you want to do it's boot a malicious image so that's a cycle um we'll see uh what you would has to offer in terms of doing

this um so yeah interrupting the boot process look for serial ports um this is a very shy job I did a couple years ago I think this was like 3 years ago this is a typical like netlink router and uh I don't know if you can see the the red border it's not it's not printed on the board but those four those four pens uh are what's the t t ins sign for a uart Serial console so you have four things one of them is ground which you need to make sure you identify I don't think you should ever sort of Jack into a board or whatever unless You' identify the ground state first because you could blow

things up and one of them is the power typically you maybe don't need to power this yourself sometimes it uses the board's power um it depends on your situation and then there's receive and transmit and uh once you plug into that you'll be able to either get a shell interrupt boot or it will just dump a whole bunch of Diagnostics to make sure you can you see what's going on um and and the bottom picture that's me just ripping like uh poor Arduino sets like jumpers cuz I didn't have anything else I I sold them on and uh hope for the best um so that's that's basic example of Serial Port the other way you can

interrupt boot um going to talk about this one this is not me this is Colin oin I think if you get into embedded stuff you you'll know who Colin oin is and this is him um sort of doing a rudimentary glitch of a Philips Hugh uh little smart bulb um board and uh what he does here is he's taking I think that's like an i j tag sort of like hitter it looks like and he's taking P pulling one of the um the there's like a ram chip on top and he's pulling that to ground the reason he's not doing the serial flash is because uh he wants the actual seral flash to work look for the

look for the other Ram trip and then fail and what happens then is he it drops a ubo shell because it's like well I can only boot from this Ram what do I do it's not there and it fails into an actual shell um so that's very simple way to glitch something I think if you're going to try this again make sure you've identified an actual ground Port um and that you're pulling it to the to the actual ground on the board it's very dangerous to do this if if if you're not aware which uh which pins you're actually using but very simple to do very easy to do once You' po the board open try try to try to pull some of the

pins to ground see if it panics and and gives you a shell um so once you've got a shell this is what it looks like this is um what I built um in preparation for the talk not very realistic scenario but uh in terms of yubo shells I think it's as real as it needs to be this is me booting uh a fresh uboot bold off of uh key the Kimu emulator for arm 64 and um what you can see here in in in the top of this in the top of the prompt is it looks for the flash it does a it does just like a normal computer it'll look for the floppy and the other boot

devices and if it doesn't find that it falls down to what's called autoboot as I mentioned and um you'll see on the screen it says hit any key to stop autoboot and at that point when you're firing up a board you might want to play with the GPI pins press some buttons and it will stop the auto Boot and you'll get a shell and then uh just because I wanted to be a badass I re I recompiled uboot and uh changed the prompt because that's the coolest thing I could think to do for the to um so yeah um cool uh it's pretty easy to do that the config uh environment options is really easy to

find so dropping ubot sh what do you do with this well there's uh typically in in in in a un sort of I'll say a naive ubot uh buold there's a bunch of commands that you might find interesting modify memory is is a really cool one probably not the first one you'd go for but modify memory allows you to just change bits in memory or bytes um if you wanted to check whether you could modify the image and then boot it this is a great one You' need to be able to find uh where your kernel loads change some bits in there and see if it actually boots it um go is great as well because

you can just tell it to start executing from a given address um you can load memory um load load meory is pretty pretty popular command and it makes part of a lot of boot scripts because it's typically what they do to make sure the kernel gets in to the main memory and and they usually just issue boot right after that there's also boot variance um boot M boot Z and boot I which you can use and the one I one one I really like is boot flow which I'll show you in a second and of course there's a bunch of options for booting from the network TFT boot is one of them is too is one of

them too so um boot flow um this is something that's quite interesting you jacked into a UB boot shell how do you find out what there is to boot like how do you find out what you can actually Boot and whether there's alternative boot paths so that's pretty interesting is there just one boot path or do they have like some recovery image that you can also boot into is it as verified as the as the main boot flow um what kind of resources um does the secondary boot use and to find that you basically go boot flow list and it'll give you a dump of like what's the boot this is obviously like a kind of fake sandbox example so

what it's showing here is the Kyu firmware um boot flow and uh once you've got a list of them you basically select one you boot and it start your kernel um you can actually see the machine tag that I prepared this on that should probably have been sanitized out um and then obviously the other way you can boot is uh Network booting so um you need to make sure your dhp setup I stole this from another example because this is a nightm setup um but yeah so you s make your D DC dchp setup you have an IP address once you have IP address you can issue tftp um load it will load into the

first argument and you give it the image name those little hashes mean that uh it's actually fetching something from the network and it'll show you how many bytes it it returned once that's done you can pretty much just issue a boot command and P your uncle stuff will start booting um the other thing you can do is check out the other interfaces so um MTD it's basically a flash um uh flash interface so it'll list your flash uh uh um cards and um you can also read from them read into them modify them and then um read out of them into main memory and boot from there so that's pretty much it and then before I finish

off I think this is one of the newer things I do I'll show you how to talk through how to actually fix a ubo instance uh according to points I think are more important so the first thing is is to disable um uh the command line and the console and um the way you do that is in your config SLC configs folder um for your your config name many of them you can just say disable console no change your boot delay um to a negative number and um the other thing you could do is disable commands so once you know what you want to boot off of just make sure you can't access your flash why

would you have Network booting on unless you're going to upgrade maybe want to upgrade the actual image after you've you've you've bured into production mode um but really cool thing is to just disable the commands you don't use um the other one is um make sure you update uh uboot uh your uboot build and your drivers um uh and then yeah the other one is number three is pretty cool I've seen people um actually hard code their kernel command lines this is great because once you Bo pop the you would shell um it means that and the way they do this is they actually hardcode the boot commands so uboot is kind of a DIY

project it's not as uh polished as other um open source projects yet and you actually you actually have the power to modify command yourself and add commands and it's really powerful to just take the boot commands boot commands would usually boot anything you want and hard code some options and if you if you outcode the command line you're eliminating a lot of risk because you can't change the way that it it instantiates the kernel um you can't change the sources it use you can't tell it to add a a shell and things like that the other thing that's really important is if you're doing verified boot you must make sure that the config uh uh

disables Legacy image formats because some legacy images do not support um verified boot so they don't have signatures and if you don't turn this off when you're booting from a foot image you can still boot legacy images without verification um obviously you can disable alternative boot flows and one thing that I think sometimes slips under the radar um especially given the amount of um memory corruption bugs we see for NFS passes foot passes and a bunch of the stuff that are uh Under the ubot Umbrella um you should make sure your compiler protections are on and um ESP if you're booting a ubot version lower than I think 2010 there's no stack protector support um so one of the cool

things you can do now is actually enable sack protector at the uood build level and make sure you you uh compile position independent as well um so that's that's pretty much it any

questions yes so what are some practical things that you marked around with that I mean I've seen some couple of talks of guys have gone off to like robotic vacuum cers and that sort of stuff obious devices those things obviously concern about what are you typically practic you Ming with so um I mean most of the time I'm uh reading configs for things like Amazon devices um I'm reading things uh things like cameras like little IP cameras are pretty cool and really old routers are very easy uh to attack as well that's typically where the devices I've uh in the past tried to look at like um USB POS predicted like USB drives so I would

try to there's nothing really running on them it's just a storage device but that's sort of the range IP cameras um uh routers and um small smaller inverted devices I think some that run minimal um socks yeah okay no other questions yes

mention I think you're just got to pop that into like a binary Ninja on Ida and try to yeah I mean you that's you got to reverse engineer that yourself I I don't know where you would get uh Insight on that besides reverse engineering yourself okay no other questions cool thanks [Applause]

everyone