BSidesCharm 2023 - Protecting Yourself From Supply Chain Attacks–Trust Is Overrated- Paul Asadoorian

foreign

[Music] hi hey thanks for coming to my talk um I like to start my talks usually talking about something kind of ridiculous and off topic and then hopefully we'll bring it on topic and I don't want you to disclose any personal health information so like if you do have high blood pressure like you don't have to raise your hand just make a mental note of it but like if anyone wants to admit they have high blood pressure if you don't it's okay because I'm going to admit that I've been diagnosed with high blood pressure um it's fine I take medication um but it's uh something that start you start thinking about some of the things

you do that might impact your blood pressure like standing up on stage and giving a presentation or drinking coffee how many people here drink coffee oh right I love coffee there is nothing like that first sip of hot coffee in the morning but now as I've disclosed I also have high blood pressure so these two things could be at odds with each other so I wake up every morning and I take my blood pressure medicine drink 12 cups of coffee and I'm the same like they just cancel each other out like what am I doing but I love coffee too much so my wife my wife that I mean people that are married right my wife loves me even though

sometimes that can be tough but uh she's like I'm really concerned because you drink a lot of coffee and you have high blood pressure and I'm like well you know if you search the internet long enough anything can become true to support well like your bad habits so some research suggests coffee can lower the risk for high blood pressure now I could probably say that really quick and it probably wouldn't get past my wife because that's only for people who don't already have high blood pressure right but we can find lots of studies that may support uh drinking coffee if you have high blood pressure um so then you know the hacker skeptic nerd and myself is like well okay so

like how much caffeine is safe and how much caffeine is in a cup of coffee and this seems to be like a very debatable thing on the internet uh roughly 95 milligrams of caffeine in a cup of coffee a little like what's a cup of coffee like my giant Yeti mug that holds almost like more than half a pot of coffee is that a cup is a cup like you know is it eight ounces in a cup because that's just like a little espresso for me in the morning because I bought this really nice coffee machine and it makes all this wonderful coffee so I drink a lot of it so like what what what's a cup

um but that also begs the question of how am I consuming my coffee how am I Brewing my coffee where am I getting my coffee from which may have different amounts of caffeine which also could have different amounts of caffeine depending on how I'm drinking it as well so this starts to question the the supply chain of my coffee how much can I safely consume now I start to scrutinize the supply chain of my coffee and wanting to measure it so I take to the internet and I go to chat GPT and I'm like I really need to know how to measure how much caffeine is in coffee now turns out the entire lecture could probably be about each of these

methods in chat GPT told me which were all super interesting right like mass spectrometry and chromatography stuff that I can't even pronounce right uh enzymatic essays right like what is all I have no idea what all this stuff is now I've read it and tried to understand it and we understand some of these methods for measuring coffee chemically separating the caffeine seeing how it reacts to light and doing different measurements they do make the little I think number three is like the little test strips that you could put in coffee it turns out because I spent more time researching this than I should have that just tells you if it's decaf or if it's regular

coffee like is there caffeine in it is it not um which is not so helpful for my use case so I was like well maybe I'll just have my own machine at home like that fun but they're kind of expensive the fifteen thousand dollars like the used version of it uh and this performs a a chemical you know a process that will measure how much caffeine is in your coffee uh separating molecules and then the incorporation of UV Spectrum I can't even say that word so spectroscopy yes thank you allows it to determine the concentration of caffeine in the coffee I'm like well I guess at this point I just have to trust the supply chain which after my

talk at shmukh Khan earlier this year in this talk and like well that really sucks Paul we really shouldn't trust the supply chain in this case I I might have to unless someone has one of these at home in their lab that I could borrow on a regular basis because I like to continuously monitor the amount of caffeine in my coffee so how do we minimize these supply chain risks I've spent at least a good solid two months as a focus on my podcast talking to people about supply chain supply chain threats risks supply chain vulnerabilities what those mean I talk with people way smarter than me and really started digging into the issues surrounding

supply chain much of which you know led to me being able to give this presentation today so some of the things that I've been thinking about recently is well how do we minimize these risks and I really think this is a scale right if we think about any supply chain whether it's coffee or whether it's software and Hardware that we all have to deal with I could create everything myself I could grow my own coffee beans which I'm not much like a plant and I really don't don't get along um so I could create everything myself I could do the same thing with software as well I could write all the software that I use from scratch I don't think many of

us want to take on that challenge today in the Middle where I think most of us land is that we verify then we trust and we work to make the attackers lives more difficult that's what I think is The Sweet Spot that yes we don't create everything ourselves but we also don't just pull everything off the shelf we do some kind of mixture of it and we validate what we get in this middle stage and so I may have all these fancy devices for making coffee right I'm not growing it myself but I'm also not going to the convenience store and grabbing a Starbucks frappuccino that's already made off the shelf right that is the

create nothing and do verification do no verification I'm just blindly going and whatever that Starbucks thing is I don't even care how much caffeine I'm just taking it and I'm drinking it now the more I think about this analogy in this slide when we advise companies today on their supply chain strategy that you're likely going to do something in all of these areas that in some cases you're going to want to create things yourself in some cases you're going to want to grab the Starbucks coffee off the shelf and accept the risk now I don't think you ever do no verification I think slowly you can verify but as we look at our supply chain uh processes in

our respective organizations you're likely going to have a combination of these things going on okay so also in this talk um it's in this Unwritten unspoken contract that I use Linux as my daily driver now uh and so this laptop up here is running Linux and it's displaying just fine okay um also these two are the the funniest posts that I've made on social media and each of these have um the one on the left on Twitter has over a million impressions the one on the right on LinkedIn has over a million impressions and I think you're they're hilarious so okay you can you can laugh now it's fine thank you thank you thank you maybe they're funnier when you see

them on social media um but it's my way of telling you that a lot of the examples I'll give in this talk will be Linux based um because that's my daily driver and I still maintain Linux systems uh that support the podcast uh so a lot of my work deals with Linux so a lot of my supply chain examples that I will give in tools are specific to Linux some of those run on other platforms such as Windows or Mac OS but largely they run on Linux okay this will be the the kind of premise for talking about the digital supply chain and again this was the result of me meeting with lots of people

thinking about the supply chain and what I wanted to create was a map on how we think about the digital supply chain because supply chain is an all-encompassing term that means a lot of things to a lot of people but let's dig into what the digital supply chain is on the left hand side we have the physical supply chain this is often what we think of when we acquire Hardware when we get a PC a laptop a server an iot device it's physical Hardware we ask ourselves we should ask ourselves has it been tampered with has someone physically altered the hardware now in terms of this happened if you search for hunting for back doors in

counterfeit Cisco devices that's where the the bottom image of that physical box is is someone actually soldered a piece of Hardware onto a Cisco device to tamper with it to install a back door so that's the physical aspect the next if we move from left to right we have what I call well what Bob Martin from might are called and I stole from Bob because I think it's a great way of thinking about it when we think about things such as firmware now firmware is just software that's inconvenient to program the other thing about firmware is that it's pre-installed when you acquire a piece of Hardware you don't have a choice of what's installed on that

laptop in terms of firmware I mean maybe you do maybe you don't but for the most part your UEFI bios has to be put together Acquired and programmed onto your Hardware by the manufacturer it's pre-installed so my second category of where we need to worry about the supply chain is pre-installed software like firmware now you also notice boot loaders are in there too but bootloaders are oblique us nerds we're going to mess with boot loaders for the most part other people aren't messing with boot loaders it's pre-installed by the operating system uh kernels also might be pre-installed and may be messed with by you know Linux nerds and such but for the most part kernels are also largely

pre-installed for most user populations then we get into operating systems we customize the operating system a little bit and that speaks to where I think firmware and there's lower level software sits is the level of customization and interaction that you have with it most users are not customizing firmware bootloaders kernels in much of the operating system okay now those of us in this room probably are doing all that stuff with all those things and that's fine but most of our users that we're supporting are not most of our users are in this the third bucket there of the digital supply chain third-party applications right when people think about using their Computing device whatever it is they're

using slack they're using Zoom they're using Google Chrome and that's also each of those pieces of software have a separate supply chain now the fourth box all the way on the right is the one where when we say supply chain it seems to have taken over this term in our industry what they really mean is the software supply chain this bucket for me is software you write yourself now nothing wrong with that uh But realize that's a separate supply chain vertical if you will where we worry about software libraries where do we get our software libraries from how do we track the transient dependencies within our software libraries how do we know how much third-party software is

included in our software that we're writing ourselves so that's the force bucket fourth bucket now notice the arrows are going from uh left to right so I think the further you go towards physical Hardware you get reduced visibility and validation is hard right gotta physically open up the hardware to look and see if there's a supply chain issue I think as you go towards uh software developed in-house you get increased customization and control I can't really control the firmware that's on my laptop I'm relying on the supply chain to deliver me that software we talked about that in the shmukon presentation so this talk is a kind of a follow-up on how you defend yourself uh

from the shmukh Khan presentation so we get that increased customization and control um though when I like to think about it as I work with firmware largely for my job today and it's always been a love of mine a love-hate relationship with firmware uh you could say that uh when I was developing software I had a lot of control I could take a library and I could go you know what I'm going to write that myself I'm not going to rely on a supply chain I'm just going to write that myself and that's going to be my library when we move left on this graph it gets harder and harder to do that I can't go oh I'm just going to

design my own chip and solder it on there to replace that that gets harder okay so that's the supply chain attack surface now when we come to the issue of trust I trust none of it I don't trust the supply chain at all I think we need to go into it knowing that we do not trust any Link in the supply chain and we have to verify every single Link in the supply chain as best we can so and the reason for that is here's some real world attacks now I purposely chose ones aside from the physical which I won't focus too much on uh for the remainder of the talk but the rest of

them are examples that happened recently like late 2022 and into 2023 these are some of the kind of poster child supply chain attacks and we'll couple these with some of the defensive mechanisms that I've been working on and researching um so I mean we'll talk a little bit these are like physical and firmware kind of lumped into one um so let's start with your TPM right um The Trusted platform module of course holds keys for things uh on your system things like secure Boot and other such things there are a couple of different vulnerabilities with this uh one is an older one um that I have I have seen detections for this is checking for the newer one

that was discovered by quarks Labs um uh there's actually two I believe vulnerabilities that were disclosed by quarks Labs um one of them is I believe a memory corruption buffer overflow inside the TPM firmware this software is where you can check for this it's super hard to find like at this level like freely available tools to check for some of these things um so I I think it's awesome and so I ran this tool on some of my systems and like lo and behold I'm I'm vulnerable I'm like okay that's great what do we do now now we've checked the supply chain we figured out oh the TPM manufacturer has given me software that's on a chip

that's vulnerable what do we do it's my best advice I'm sorry I don't have great advice for depending defending the supply chain so you know there might be another talk going on uh but uh that's set it on fire and so here's uh here's the story behind this is that uh Yvonne Arcee uh and I go go way back he's the founder of uh core security Technologies from back in the day they were the first sponsor of my podcast um so Yvonne works for quarks Labs Yvonne is the director and coordinator for the vulnerability research at quarks Labs it was one of the researchers on Yvonne's team that discovered these vulnerabilities in the TPM so Yvonne

worked with the researcher and worked the disclosure process for this in that disclosure process Yvonne goes you know let me check my own uh Dell laptop okay now I'm not picking on Dell it's even playing field out there right it just in this case it happened to be Dell variable could have been HP Lenovo or whatever but just happened to be Dell let me check my Dell laptop turns out he checks his Dell laptop the TPM is vulnerable to the vulnerabilities that his researcher found so he goes to the chip manufacturer and he says look uh this this Dell laptop like on your chip it's vulnerable and they're like well that chip that runs the TPM is into life according to

the TPM manufacturer he's like okay he's like so you're not gonna make a fix for it and they're like no he's like but my laptop is still supported by Dell so you should expect that Dell is going to go to you and ask you for an update and when you tell Dell that the chip has been uh discontinued they're probably not going to be too happy about that or maybe I don't know I don't want to represent Dell here so when you go to Dell for Yvonne's laptop there's still no update for his TPM so that's why like set it on fire is the really only way right now to kill that vulnerability otherwise you're just waiting on

Upstream uh supply chain issues so it's good to have the tool to check for it but we also need a plan on the back end right now you got to start thinking about replacing that Hardware updating it if uh or using it in a different manner right depending on the sensitivity level uh of the the data or the Personnel involved in this laptop so these are the kind of supply chain problems that we deal with first step is Discovery and pressuring oems and manufacturers to get these fixes out there um another great tool is lvfs and FW upd it is a suite of tools for Linux and it allows you to do a couple things one is

enumerate all the devices in Hardware within your system right there are ways to do this on all all different platforms I just happen to focus in on on the Linux one this is a great free and open source tool two components it was created by Richard Hughes and the two components are the lvfs which is the database of all the firmware that works directly with Hardware manufacturers and vendors they upload their firmware updates to lvfs the Linux vendor firmware service FW upd is the open source free and open source software on Linux that allows you that gives you the set of utilities to enumerate the hardware on your system enumerate the firmware on your system

and also apply updates to vendors who are participating in the program so when you're looking at your hardware and considering the supply chain this is a great thing to use the other Tool uh I thought I had another slide okay that's fine um the other tool that I found uh and this one was kind of interesting because we were really like figuring out ways on Linux I want to know what audio drivers and audio stack that I have and getting that answer easily led me and I stumbled upon this Tool uh called inksy I'll call it inxi and it does an amazing job of clearing all the hardware on your system and giving you a complete list now if

you just run it by itself it gives you a little bit of output you can manipulate the command line output to give you a lot of information telling you which Hardware you have so you can start evaluating the the supply chain threats and or risks that you may have one thing that you'll notice that I'm kind of I'm kind of proud of is that's my desktop the 24 core threadripper it's awesome it's awesome uh and when I built the system uh probably two or three years ago now uh I was like well I'll upgrade it eventually and now I'm sitting here two three years from now going yeah I still can't find anything that's like will be much faster than

what I have now like that's pretty awesome it's got 256 gigs of RAM uh which is impressive now I don't I don't use all that power I just it's bragging rights right so so there um but also what that means from a supply chain perspective I built this massive computer that I use it as my desktop uh which I love but I'm going to have that for like a long time right like it's not going to get slow or I'm not anytime in the future which means I have to look at and preserve the supply chain of the hardware and firmware components that are on the system the manufacturer may stop and say I'm not giving any more

firmware updates that is uh end of life right now sorry you're out of luck I'm not lighting this computer on fire right so we've got an interesting situation on our hands the bigger systems we build the most the longer we have to support it I think for me one of the takeaways is to start looking into some of the open source firmware like core boot My Hope Is we've seen some evidence of this as core boot supports more Hardware open source will continue support where the manufacturer left off and I think that that is my tip basically for for this situation and the TPM is a harder one because there's probably like licensing and other mitigating factors but for

things like UEFI for example core boot can come up with open source Alternatives where the manufacturer has left off and left you vulnerable okay let's look at some pre-installed um firmware uh in Events first so the MSI breach and the supply chain has everyone heard about the MSI breach uh a few of you have so uh MSI was a victim of a ransomware attack now look I'm just going to tell you like what I've read I'm not claiming any inside knowledge so like please don't get me in trouble or stop the recording I can tell you more um but they're the victim of a ransomware attack the ransomware threat actors stole 1.5 terabytes of software including source

code bios development Frameworks and private Keys that's one of the dangerous things now we did see Intel had some uh source code and keys leaked turns out that was for boot guard turns out those keys were not pushed out to production systems they were only using development as far as I can tell uh MSI on the other hand uh I believe that these some of these keys are used in production this essentially means that the entire supply chain for MSI hardware and any hardware related to the leak is potentially easily compromised by attackers this means they can create their own signed code to run on your Hardware it is the the cryptographic signing of this

firmware that prevents attackers from doing this but once all the source code leaks in the private Keys leak it's it's you know the gloves come off and anyone can produce a firmware update that firmware update will be presented installed on your system it'll check out and that means an attacker now has persistent very high privilege access to your system that is very very difficult to detect um so we also I didn't did I say we I say also it was published that uh MSI systems had a lack of signature Updates this is something we've been uh pushing for is the update package is actually signed so that you're not installing just anyone's update to you EFI system

that you're installing the assigned update from the manufacturer some manufacturers don't sign their updates MSI was one of them since all this stuff leaked we could go through the leak and go did you sign any of the stuff no okay well then uh Eclipse team has a couple of blogs that are summarizing everything that our research team has found so far Black Lotus is another instance and this one this this one I particularly love because it's a it's an interesting aspect of supply chain in that like I'm bringing my own vulnerable modules or code to the system I've been seeing a trend as I cover the articles on security weekly that once an attacker gains a foothold if there's not enough

attack surface that's vulnerable enough they'll just bring their own PHP module and put it on there that's vulnerable to your WordPress server they'll bring their own bootloader in in the case of black lotus and install also has a vulnerability they can exploit to increase their privilege level that allows them to be more persistent which also gives them greater impact on the system as well that's essentially black lotus in a nutshell um also slides out order so we're going to switch back I explained all of the uh after all so the other part of fwfu the reason I put this in here is there's another feature so not just enumerating your Hardware so now we're talking about

there could be vulnerabilities and conditions in your UEFI firmware that make it ripe for attackers well FW upd has this great Little Hardware security module um that you can run and what's interesting is uh it hasn't been widely publicized and basically it scans your hardware and firmware on your system and classifies your system into different what he calls host security IDs or security models so if you fail enough tests it's basically a score he's like you're not at a good security model for your firmware which is awesome uh for you to know not awesome because then you have to fix it but uh it's a great little so it's FW upd MGR security um I used to use dash dash Force for

certain certain Hardware that it may or may not recognize uh and it gives you a basically a readout of not just what firmware might exist but what configuration and other options might not exist on there as well like Intel boot guard uh you know is it is it there is it enabled um so this is developed by Richard Hughes uh he works for Red Hat uh I did an interview with him on the eclipsing podcast so you can listen to our entire discussion uh about how the project started where it's going and all the things you can do with it but if you're looking to validate the supply chain did your Hardware manufacturer give you

hardware and firmware that was in a secure State well one of the ways you can test that is using free and open source tool secure boots also like a really good thing to have uh as well and I still feel like there's some misconceptions out there and maybe not with you folks in the audience um but with the community like larger Community uh some people think secure boot is well that's just a physical mechanism to protect it sometimes when I talk to administrators they're like well you know yeah I can enable not enable but that only protects me if someone has physical access access to the system I'm like no secure boot is cryptographically verifying every piece of software in

your boot process it's a great protection mechanism against supply chain attacks because if there is a supply chain attack that doesn't compromise secure boot then we have those as well that's the last bullet on the slide and that happens however if there are other supply chain attacks that are not bypassing the secure boot mechanism it's a great way to ensure the Integrity of your system however it hinges upon the fact this is a theme that will come up that you keep your revocation list up to date revocation list updating is a Hot Topic it's not something we directly have control over we we can take that back we can if you want to assume all the responsibility

for maintaining all the keys and cryptographically signing every piece of software on your system Intel and Microsoft develop secure boot so you could do that that's a lot of work and responsibility so I don't necessarily recommend that however then you're relying on the oems and manufacturers to update the revocation list because they control the keys and it's up to you and us in the community to make sure that it gets updated um lvfs again FW upd has done this for Linux systems you can do this in Windows um apple and Mac OS have their own process for that because they control the hardware firmware and software more than any other manufacturer so we basically just have to trust it like

Apple's good which we can have that debate however if we go back to the Linux systems um Richard did a great job of updating FW up D to help you update your dbx on Linux because on Linux I still think it's difficult to detect whose responsibility is this to update the revocation list for secure boot should it be should it be Microsoft should it be your OEM or manufacturer through a UEFI update should it be your operating system vendor well if you're using shim and shim is shared by multiple Linux distributions whose job is it to update all of that software and all those revocation lists so Richard said you know what I can help you with that and

it does a great job of doing it it Flags situations such as if you're going to update your dbx I was trying to help a random person on the internet with this and they just weren't getting it because they were trying to update their dbx and like the updates failing and it says that my my bootloader is in the in the dbx I'm like yeah that's because you haven't updated your bootloader and they're like no no it's up it's up to date I'm like no no the hash of your bootloader is in the revocation list FW upd is telling you that if you apply this update when you reboot and you have secure boot enabled it's not going to

boot because your bootloader is in the dbx it's warning of that before all those bad things happen so go update your bootloader he's like I'll just I'll wait for an update to FW upd I'm like no no never mind hopefully they figured it out okay let's look at some third-party software how are we doing on time good 3cx was that a train wreck or what good Lord the more I looked into this so now we're talking about third-party software right this desktop software that people use and we've all probably got like little Snippets from the story you know that we pulled so I'm gonna go into too much detail here but what was astonishing to me was how much of a

train wreck this was a a supply chain attack that led to another supply chain attack that compromised software that was for voice over IP for for call centers and in places like that they got pushed out to all of the users that did some side Channel loading dlls like there's a this is an amazing you could give a whole talk just on this whole thing right here but this is most definitely a great example of third-party software in the supply chain being compromised the vendor also handled it very very very very very poorly it was really really bad they're like there's no there's no problem here the like Jedi hand like is no problem here your EDR must be as a false

positive or something like no your entire supply chain uh has been compromised now and we're still uncovering the extent of these damages because the ax Trader software was compromised in a supply chain attack the 3cx builds were compromised in a supply chain attack uh and so there's others being affected by this breach and it just keeps unraveling and it's a hot mess um so I want to go uh to a Linux example of third-party software and it's interesting this is what set me down the path of getting really interested in supply chain stuff you know my day job was getting more interested in supply chain stuff and immediately when they did that like I I started relating it to

me right I try and make things relatable uh to my own experiences and I was like you know I used I use Linux again I'm obligated to tell you that's in the unwritten contract and on my desktop so I'm like I update all the software all the time and I used to be I'm a recovering Ubuntu user I used to be in Ubuntu user and what what got me with the whole supply chain issue there was I'm running Ubuntu and I want to install software and if I use ubuntu's package repos it's either not there or it's really outdated so I'm like all right all right fine okay I'll go to snap like maybe snap and then I'm like but there's

like two different snaps out there and then snap does a sandboxing that breaks the function functionality of the snap and then but now I also have to remember to update snap I'm like oh okay that that's fine uh maybe I'll just go to the uh like Google Chrome has its own repository and I'll go to their repo and then I got to make sure I trust their keys I gotta validate their keys and now I'm getting Chrome from them and if I want Firefox I get a disable Firefox and ubuntu's official one uh and then what is the other now I'm having a mental blank what's the other Linux uh one that they want to use that's like Snap what

is that called flat pack thank you then I'm like oh then we have flat pack but what strikes me about all these different areas is I'm like someone else is packaging my software and do I trust them or not and so I switched to Manjaro and Arch Linux and I still have the same problems but I'm not running Ubuntu anymore uh so here's now I validated this with a member of the the Arch Linux security team um who happens to be one of my new friends that came on the show and I was like dude this is really bad and he wrote me a very nice email Santiago's an amazing person and they're working on

this on this issue he's like I hear you he's like we know about this and we're working on resolving it which breeds some confidence in me they're not just like sweeping it under the rug like some some vendors do right but basically uh this article described a situation where they're able to pull off this attack where in Arch Linux packaging you as the packager can turn off signature validation for your own software now I believe this feature was put in there if you have something as part of your package that you don't need to verify uh as part of it maybe it's a glob of data that has no security implications in these weird use cases this is what

happens when you make software for everyone you could account for all these different use cases I'm not a fan of the feature because what I discovered is when you're as the end user updating the software the package maintainer goes yeah we're just going to skip validation for that it doesn't give me an option to go do I accept or not it just continues on with life and I'm like I I don't I don't like this and I think this is one of my you know this is one of my tips for all of you in validating and defending the supply chain is questioning this stuff you have to really look for it when you're doing

things like updating packages look for it and reach out to people right so now the the team is looking at this um and so what happens is if you skip that validation the other thing you have to do is find expired domains and you have to take over the domain to deliver that software and since you can do that and the software is not signed you can install malicious software on people's devices um and it's all explained in that article but again this is one of the supply chain things that uh I come across uh and it's not the only one right um when you're updating as well it may ask you to validate a key and this is

where I think we just need to be extra diligent and this has been a problem ever since we started signing software in any capacity now there's a you know a check sum and a signing process those are two totally different things but in this case I was installing software like I want to run Spotify on my Linux desktop I don't want to have to build Spotify I don't have to like go out of my way I do want to have some trust that there's a third party that I can trust that will validate this will produce a package that can be validated however when I'm updating it says do trust this key now be honest

how many you're laughing already how many of you when you're in a hurry you want to listen to some music you got a lot of meetings that day and project deadlines are just going to click yes like just yes yeah come on be honest it's fine it's fine it's fine right what I'm proposing is that we notice situations like this that we work with package maintainers that come up with more automated ways to validate the key which is being worked on um so that we just don't click yes right and I think this is part of the if we're going to start to have solutions for the supply chain we need to not just be in

the mode of clicking yes but also the update mechanisms need to help us as well help us do better help us more easily validate all these things that aren't so manual okay let's look at software developed in-house I love this paper this might be my favorite security paper of all time we've We've joked about it on on the show actually because when we were talking heavily about supply chain Josh marpett would be like you'll remember Paul there's the paper that talks about the compiler inside of the compiler inside of the compiler I'm like I know Josh I know and then like I got it wrong and who wrote the paper but it's Ken Thompson's paper Reflections on trusting

trust in August of 1984. I don't know how old all of you were in 1984 um but I was I was in elementary school and Ken was right in the smartest paper in the world on supply chain security does anyone know how many pages it is how many pages how many did you guess eight Jeff I know you know put your hand down eight two it's three it's three pages I was a star I'm like oh my God and so you read it and then you read it again and then you go I still I still don't wow okay let me put it down and I'll go read it again uh but he he really just he's

summer is so densely packed and so eloquently just summarizes software supply chain security you cannot trust code you did not totally create yourself I mean it's of course like that I'm like oh well if we'd listen to Ken 1984 we wouldn't have the supply chain problem but we got lazy we rely on other people's code uh I mean from the beginning of time we rely on people's code there were Punch Cards in a drawer in a central location that people used to share so we've been sharing code ever since we could write software on Punch Cards um no amount of source level verification or scrutiny will protect you from using untrusted Code doesn't matter how much we look at the code

finding logic flaws and other things is super super hard as especially as we have millions and millions of lines of code and roughly estimates I see that 70 percent of that code is open source in someone else's software um so I encourage you to go back and when we think about software security read Ken's paper and then read it again and I just be astonished that it was written in 1984. so amazing stuff so we started applying some of those Concepts I am uh working on I don't have a date when it's going to be released but I'm working on a class the class is centered around vulnerability management my first task was to set out and build a whole bunch

of vulnerable stuff you'd think that would be easier than it turns out that it gets super hard like why is this so hard putting up vulnerable stuff like I used to it would be like oh my God it's vulnerable it's really horrible and I'm like well I really want it to be vulnerable why is it so hard um so but one of the things and this is like in order to have supply chain Security in your software you can do the opposite of what I'm doing here so it's like learning by the opposite in order to have a really good reliable vulnerable system I had to build a lot of things myself like literally in

Docker from scratch so I couldn't rely on the Upstream because the older stuff to their credit Debian and others like they're taking that stuff down like they don't want you to use the older stuff so a lot of that older stuff will disappear and go away but I'm like no I need that I but put that put that back so one way to solve the problem is I did from scratch which basically gives you an empty container the next thing you need in that container is the root file system I stole an old root file system from Debian eight I think Jesse is Jesse eight I think it was Jesse was eight or nine uh there's a Debbie and Jesse uh

build root and I install that now I've got a reliable way no one can Muck with my build route because I downloaded it I have it when I build the base container it puts an old version of Debian in there and then I can go in and put my stuff that I want to be in every one of my vulnerable containers inside of that and I build it myself if you've ever and then you know I looked at all of the vulnerable scripts for containers uh to create containers that are vulnerable and I re-roll all of those as well so they were more they were vulnerable in a more reliable fashion and uh so I had this really really great

environment but now this was to be vulnerable to not be vulnerable to go back to the Ken Thompson paper build stuff yourself I see so many Docker containers that are just horrible level security because they're just willy-nilly pulling in components from wherever and not validating so I think if you have a chance to look at containers in your environment you should go towards building more things on your own don't just do from you know whatever PHP application from WordPress because inside of that is someone else's code it's downloading all their people's code and installing on your systems so try and lean more towards building your own things you might probably have to go from this extreme because this was from

a vulnerable lab but definitely go towards the more build it yourself when you're working with containers another way to deal with specifically containers and also libraries is Google's got a pretty new a new and pretty cool project where right now it's Java in Python they are producing software packages that are validated by Google so I mean if you trust who do you trust do you trust Google do we trust Google they used to say they weren't evil and they stopped saying that so I don't know if I trust them as much but I do think this project is cool they produce s-bombs in spdx and vax formats they are applying uh salsa artifacts for that these are all some of

the newer projects that are talking about the supply chain so you can get yours if you have to rely on the I want to go get a coffee and I want to just pull it from the convenience store and drink it get it from Google because they're doing a higher level of assurance I think this this project is they're saying all the right things to me so hopefully this is successful so in conclusion uh Hardware firmware third-party software application uh software developer strategy based on some of the suggestions I have here have a strategy for each of them right though they are four distinct areas of the supply chain attack surface one other thing that I really love as advice is

you need to monitor for changes continuously that is my best examples I have ironically enough in MSI laptop this is not an MSI laptop this is a framework framework did a great job by the way with their with their firmware but MSI did not and MSI stopped producing a you if I bios update pretty much right as I bought it in 2020. so what I what do you do as I said you can let it on fire you know your TPM example from earlier what I do is I use software to monitor any changes in my UEFI environment if that changes it shouldn't change if MSI hasn't released an update you shouldn't change if it changes an

attacker may have modified it I do the same thing with my boot loader if my bootloader changes as a result of not an operating system update that I applied I want to know about it so I monitor my boot loaders and go if that changes I want an alert and I want to cross-reference with my operating system updates so not only getting a good supply chain from the beginning but monitoring moving forward is important s-bombs I like s-bombs a lot but I think it's the s-bomb comparison that's really where it's at the manufacturer of whatever component is producing an s-bomb great take that then analyze their Hardware software firmware and create your own s-bomb and then compare

them and continuously compare them over time I always like to thank my co-workers and others that have worked in firmware longer than I have I learned everything about firmware from Reading their research and talking with them I work with an amazing team at eclipsium and I have to give them credit because they make me sound smart and at the end it's just some resources pretty much same ones from my schum presentation they kind of laid the foundation from some of the more technical firmware stuff that speaks to the supply chain so I don't think I have a slide after that that was it so thanks for coming and we'll take any questions do we have time for questions thank you

thank you mold I can't see us there's one minute can we go one minute for questions [Music] yes

yes correct

so the question is where do you start fixing hardware and firmware specific issues so uh in my schmoocom presentation you might have seen the Spiderman meme we're all Spider-Man's pointing at each other yeah and each one is you know AMD is a Spider-Man and Microsoft is Spider-Man uh in in that's and that's the challenge where where do you get you're gonna have to again sometimes open source is going to give you an update sometimes Microsoft is going to push an update for you so if it's a component so now we're stepping outside of UEFI like let's say it's some other kind your EC firmware Microsoft might work with that OEM to push it out and there's multiple different ways to

get firmware for all of those different components which makes it hard if that's really bothering you and you can standardize on a platform it's what gives like Microsoft Surface and Apple Mac OS in a real Advantage because they control more of the hardware software and firmware so like my Surface Pro I have a so confession your Linux desktop user you have an emergency Windows laptop guilty mine is a Surface Pro 3 in firmware wise and supply chain wise it might be the most secure device that I own because every time I do a Microsoft update a Windows update it applies all of all the firmware for me so but when we get into the larger ecosystem yeah it

gets complicated it gets complicated that's why efforts like lvfs are working to have a central place to put all those things so we can get updates from a central place yes Richard

yeah the question is uh where where do we start with the supply addressing supply chain issues do we start with Hardware firmware do we start with third-party software uh I'd like to start with the base layer I like to start at the uh Hardware level and the firmware level and reach a place and there's multiple efforts on this lvfs is working on a method uh and there are other things in the works that I can't talk about yet with the goal being something that I talked about as well is I want to go to a place and say I want new laptops I want 20 000 new laptops what about this Dell model Oh look The

s-bomb is pretty clean the firmware looks like it's in good shape because lvfs already scanned the firmware so I want to start in a known good state so we are developing ways as an industry that before you purchase the gear you'll be able to glean some information about the supply chain of that device and that is definitely where I would start and then when we get into third-party software and all that stuff you know then it gets trickier how we validate that on each of those but if you're not securing that hardware and firmware level an attacker that gets on that system could could persist and gain privileges yes

I've not looked into bootstrap compilers it the term sounds familiar yeah we didn't talk about the what was in Ken Thompson's paper is the it started with a writing a program that can reproduce itself and that led to what if I put something in the compiler they can backdoor login.c but then someone could detect that so the compiler that compiles the compiler will put the back door on the compiler so that any program has has that back door and bootstrap compiler sounds familiar but it might have something I read in in passing we'll talk after any other questions we have next speakers here are we good I'll be I'll be around for a little while Jason

one last question Jason 90 milligrams on average foreign thank you [Applause]