From Boot To Root: Identifying And Mitigating Security Issues In Bootloaders - Richard Weinberger

So thanks a lot for having me. So I thought maybe you have never seen a lead-ups before. These are so-called living stones or lead-ups. I like to grow them. So now you know what my hobby is, beside of computer science. So a few words on me. My name is Richard Weinberger. I am a co-founder of a small conducting company located in Innsbruck. Our main focus is Linux and security. Basically, we have a lot of engineering background, but during the last years, a lot of security since also Linux embedded gain security over the last years. And that's why I also had lately a look at bootloaders. And today I will share what I have found so far.

Let's start easy. Bootloader 101. What is a bootloader? The bootloader is started by the boot ROM, for example, on the x86 architecture. It's the BIOS or the UEFI that started the bootloader on a better systems. It's usually the boot ROM that is part of the system on chip, starts the bootloader. And then the bootloader starts basically the operating system being, it loads the the kernel binary and then the kernel does the rest. And of course, sometimes you have fancy UI or not. That's the basic thing that a bootloader does. Now, real quick, when you are in front of a regular Linux system, how to get root on that? Just open the bootloader config, change the kernel command line to init equals bin h, and

you get immediately a root shell. That's easy. Of course, what is the solution? We need to lock down the bootloader. We need to make sure that nobody can change the bootloader configuration, either from the inside or from the outside. So on a typical Linux system on a laptop, for example, you can protect bootloader config using a password. So to change the configuration at runtime, you have to enter the password easy. But when I open the disk, then I can always change the configuration on the disk itself. But for that, I have to have physical access. So, as I said, bootloader lockdown, on a security constraint system, bootloader lockdown means there's absolutely no way to change the boot config. There's no way to get ever

a boot shell. So, for example, on the U-boot bootloader, you usually have a boot shell. You must not have this. And every input has to come from either a trusted source or it has to be fully authenticated. For example, when the bootloader is loading the kernel, then it verifies first the signature of the kernel file. And when the signature is not valid, the boot is interrupted. This all sounds easier than it is, of course. Before we cared about security on an embedded system, everything was easy. So the bootloader just had to load the kernel and all good. Now we have fully verified boots where they have a so-called chain of trust. That means that only systems are

loaded on your computer that are signed by you because you want to make sure that only your software runs and nobody can change the software. For this you need special support by the hardware, but most systems these days support this. So the boot ROM knows that it must only execute a bootloader that is signed by a certain key. Then the bootloader itself also has access to a public key and make sure that it's only loading a kernel that is signed by a certain key, and then the kernel does the rest. This is really common, not only on smartphones, also on Linux embedded in the industrial field, and it will be much more and even mandatory.

So hello, Cyber Resistence, Cyber Resistence, hello, NIS2, so it will be much more. The problem is, most bootloaders we have on the Linux embedded area have not designed for this. They are really great and awesome, they work super, but they were not designed with with security in mind so when you can break into the bootloader then you basically get immediately root because then you can execute whatever you want and you can change the configuration and depending on what the device does you can also fake whatever that device should be

so now i get into uboot and bearbox who has ever worked somehow with linux embedded has seen the U-boot or the barebox bootloader. They are omnipresent. I guess so, yeah. So when you get whatever port-a-port package from a vendor, it's either U-boot or barebox. So they are really nice and cool. They're extremely common. They have also support to support it with verified boot, so they can actually check the signature of the kernel, the signature of the boot environment, the content file, everything. Awesome. the company we work a lot with linux embedded and also with security i thought okay let's have a look at the at this bootloader with my security head on not with my linux embedded engineer head on because usually what i

do with you put in that box is just get them work on the new hardware and that don't look deeper but i thought okay now let's look at the security things i thought what are actually the The critical paths, which paths are interesting from a security point of view when a lockdown system is booting up? So basically, the config file parsing, so on U-boot is the U-boot environment, also parsing out of state, for example, the boot counter stored in the EEPROM, and also another critical path is loading the boot files, like kernel, device tree, init RAM effects, All of these outputs are already authenticated. So when you change these files or configuration for whatever purpose, then usually the bootloader will notice immediately before it's starting passing this input

because the signature doesn't match. But there's one elephant in the room, the file system. Usually all the files you load from the bootloader are stored on some kind of file system. And this is the problem. So the data on the file system is usually identicated. It's perfectly fine. But the file system itself is not. And who has ever looked at the file system knows that the file system like ext4 is everything but trivial. So understanding, parsing, and making sure that these data structures are correct is not easy. And most

file stem drivers in bootloaders have been designed to be good enough to read a file and that's it they have not been designed to to work with a hostile file system so when an attack when an attacker can change a few bits on your on a flash in the old days the bootloader just crashed and the boot was over but it was your your own fault but now when you have to to make sure that everything is verified things are interesting So, and the device that can get manipulated by an attacker is a real thing. It can be done offline and also in some cases online. So when an attacker can get access to your device and can whatever to storage, then he can also

change the device stem. So now I'll show you a few of my findings. So the most interesting or funny or sad ones, The first thing is I had a brief look at the Xt4 implementation in U-Boot and Bearbox. Both share the same device, so the same driver for Xt4. And I found an integer flow in the code that is handling symbolic links. So a symbolic link is just a reference to another file. So when you read the link, you read the the next part, and here the driver is basically reading the similar contents. So first it looks at the EXE4 data structure, how much space should it allocate. It's reading this from the EXE4, adds one more byte, because for the null terminator, EXE4 makes sure that the

data's not stored with the null terminator to save bytes. So you would just, takes the length from disk as it is and adds one. What happens when I store on disk a length of FFFF and add one to it? Then we have an integer overflow, it turns to zero. Then the bootloader does see a log of zero. A zero allocation is perfectly fine. It returns not null. You get a buffer of size zero. But sadly, then the bootloader continues reading data of size DIRN or INO size into the zero size buffer, and they can override the heap. This works on U-boot and bearbox.

The next bug I found is the very same integer overflow in SquatchFS. Luckily,

No, actually it works in barebox and uBoot. The fun fact is regarding SquatchFS, uBoot and barebox have a different implementation, but they've both made the same mistake. It's a thing. Then I found another one in the SquatchFS implementation of only uBoot. Symbolics are fun, by the way. So you can have a symbolic link that points to a link, to a link, to a link, to a link. And usually at some point you have to break the loop. This was forgotten in U-Boot. Here you see the code. Here you have the SquatchFS file called FS size function. And when it's of type symlinks, it just runs the function again. So here you have a beefy stack overflow

that can be controlled by the chain length of our symlinks.

So you just have to place a long enough chain on the SquatchFS and you can smash the stack. After implementing fixes for this bugs, I found that my exploits still work. Then I found out that the memory allocator is broken. So I found multiple integer offloads in the memory allocator. It was a little bit strange because they are using dark layers malloc. This is a well-known memory allocator that works fine and is proven to be correct, but sadly, 25 years ago, in the original U-Boot code, they broke it. The bug is, you can here see in the malloc implementation, they get in the number of bytes they should allocate, then they do bedding to the weaker size, and in this bedding function, they cast into a

long. take the request size at your bedding and when it's long you have again an integer overflow so you you ask for a gigabyte of memory but just get a few bytes and that's of course also a beefy bug form to overflow then i found in this code another overflow in the in a segmentation break function and i found out when you run you put on 64-bit intel then the point difference type is too too small when you allocate more than four gigabytes you can also again and trigger an overflow in my log that was my face at this point yeah so that just by looking a few days at the first implementations i found actually at

least four beefy bugs that can be triggered on a system just by flipping a few bits on the file stem. The good thing is I send bug reports and batches and U-Boot and Bearbox guys were really happy and merged my fixes and the next U-Boot release will also contain my fixes and while testing I fixed also the U-Boot ASAN so the address-sanitizer integration. Now you have a better chance with fussing to find bugs.

One might now think, okay, I have a secure boot environment. Thank you for the bug figures. I will just update, easy. What most people forget is downgrade attacks. Imagine you have a bootloader binary that is signed by your key. So every device that has your signing key will boot this binary. And this binary is exploitable. So any attacker can always take this old broken binary, flash it on this system, the system will boot it and then they can exploit it. That's why you always have to have a way that you can avoid downgrade attacks. So you have to make sure that when you have once booted a newer release of a boot order, that you will never boot the old version. This is not so easy.

You have basically two ways to achieve this. You can have a rework list. So basically you throw your own signing key away and deploy a new one that's super hard and almost impossible. Or you have in the hardware an encounter that is identicated and goes up whenever you install a new version that has proven to be correct. This is what most smartphones do. but implementing this on a regular embedded system is not so trivial and is mostly forgotten.

I was talking a lot about Linux embedded and how awesome U-boot and webbox are, but you may think, yeah, Intel is fine with Grub. It's not a big deal. Yeah. So most of the bugs I have shown have been already found in the Grub bootloader. They have also their own implementations of file systems.

I have now said a few things on what is broken, how to fix it, but how to improve the situation. The main problem is that bootloaders have trivial and simple implementations of file systems that have been created ad hoc. And parsing these structures is hard. What can we do? For example, Bearbox is following a new strategy. They try to reuse the kernel implementation. For example, they're using SquashFS from the Linux kernel. I tried one of my modified SquashFS images on Bearbox and Bearbox crashed. Then I thought, okay, they're using the Linux implementation. Maybe I found a bug in the kernel. Then I tried the very same in the kernel, the kernel did not crash. And I

thought, how is this possible? Barebox is using the kernel implementation. A brief look at the GIFT history reveals that this kind of bug was fixed in the kernel many years ago. Barebox just forgot to backport the bug fix. So doing this is also not so easy. And the problem is also when you copy the first implementation from the kernel is that These implementations support read and write. So this is a lot of code size. And they also expect the whole kernel VFS, the virtual first time switch. And this is a non-trivial interface. For example, one thing I didn't know, I talked with the Bearbox guys, is they also do fast testing. But they say whenever they start fast testing against their barebox, then they run immediately

out of memory because the fuzzer is allocating so many files and these file handles are never freed because the problem is in barebox, they're using the Linux implementation of SquashFS and the Linux implementation assumes that the file handles are freed later per read copy update by a second thread. and this thread is not existent on barebox. So they basically have to duplicate the whole VFS implementation of the kernel, and this is not doable. So, and the toy implementations are mostly broken and copied, and some people say, just throw away the bootloader, just use Linux as bootloader. You can use function as a single k-exec, but this is also problematic, and One idea is that you have a same minimum implementation like libfsx4sqf4

for bare metal that is good enough for bootloaders, but somebody has to implement this in the same way. It has to be synchronized with the kernel, and the problem is also about license. So far, the best way we have right now is reviewing the bootloaders, make sure that they can be fast tested. especially Bearbox is doing a really great job with fussing. They are aware of the problem. Of course, we can rewrite it in Rust, but hey.

And one thing I would also say, when you implement a system that relies on verified boot, make sure you use a very simple file system. So the only file system I have only found out of bounds read what the FUT implementation in newbult. So use FUT. Don't use something fancy, use FUT. Have a boot partition that is simple and stupid, or even, when possible, use no false system at all. Have a few small partitions where the boot order can just read the data directly. That's also perfectly fine. So I'm almost out of time. So thanks a lot for being here. If you have questions, just ask me or ask me later. I hope you have had equally fun than I had.

Any questions?

The other question.

That was amazing, impressive work. Thank you so much for that talk. Have you managed to code a full exploit chain, especially with integer overflows? No, I didn't do that. What I did, I created for each bug a crafted file system and I found a way that I really can smash the stack, for example, or I can control the injunction pointer, but I didn't implement a full exploit. I have a second question if nobody has questions. I heard about UFI bootkits and also the APTs use them in the wild, but I haven't heard of like Linux bootkits, especially for embedded systems. Do you know about attackers are using or targeting them in the wild? Yeah, they attack them in a different way. Usually they

they try to find a way how to break into the bootloader that they can then extract from the bootloader, for example, the disk encryption keys. This is mostly the target. Because the bootloader has to set up all the disk encryption and it's loading from another location the keys. And when it can control the bootloader, then it can access to the hidden data on the device. True. So we are not talking about persistence like in Windows and Windows? No, no. It's just stealing. how to break this device that can extract whatever information is stored on there or that you can install what software you ever want. Thank you. Thank you.

Yeah, thanks for the amazing talk. I was wondering, so you were finding bugs in the files and some drivers. Yes. But how do you exploit them if the file system is signed that I did not. The file stamp is not signed. That's the problem. It's only the data on the file stamp is signed. The file stamp itself is not signed. So on, for example, Android, Linux is using DM Rarity or DM Integrity. There you have no such attack surface because then you would notice, but at the boot size you have nothing. That's the problem. Okay. So it's just the files on the file system that are signed and not the entire partition. Exactly.

Any other questions?

Again, thank you for the talk. You mentioned that, for example, there is a problem where the attackers may install a previous version that is vulnerable. So the solution would be to reject any previous versions from happening again. Exactly. But you have to have a boot version counter for that. But from my understanding, there is always a possibility that the updated version is not working as expected. So sometimes it's necessary to do. Exactly. That's why you can only increase this counter when you're very sure that the new version is perfectly working.

Okay. Yeah. Thank you for the answer. So I know that's nasty and not so easy.

so thank you richard another question um the sock vendors like silings have proprietary mechanisms where you can save some keys and diffuse some and do some checks before this can you say something about this mechanisms and how they deal with uh system stuff yeah so um some socks as supported to install and key material into the EVE users and then they will only boot a bootloader that is signed. But when you can exploit the signed bootloader, you can also execute code as you want. That's the main problem. So all the nice crypto in the hardware will only make sure that the bootloader that is loaded is signed. But when you can exploit the signed bootloader, then the game

is over again. Thank you.

See any other hands. So thank you very much. Thank you.