A New XZ Every Day – The Collapse of OSS Supply Chain Security

All right. Hey everybody. Um, thanks for coming. Um, I've got a lot of stuff here, so I'm gonna move pretty fast. I had to cut this down quite a bit last night because I didn't realize how much stuff I have. Um, but I'll be around afterwards for Q&A. I'm going to probably go to put my badge together in the little soldering room. Um, but I'll be floating around for sure. Um, if you want to see the slides, this QR code will take you to my GitHub repo. Uh, you can download the slides there. And actually, those slides have better, they're better than these. They have all the speaker notes in a separate breakout, so you can read, you know, you

don't have to remember everything. Um, my name is Paul Novice. I work for a company called Hunted Labs. We do software, supply chain, security, consulting services products and stuff. Um, I'm not on Twitter, but I'm on the Masttoon or Signal, whatever, or just email me if you have any questions afterwards. Um, agenda, we're just going to talk. There's a couple of pieces I'm going to set on the board first before we get to XZ and then these types, targets, tactics of supply chain attacks and how they're changed kind of recently. Um just real quick on me just so you know this is going to tell you about my biases. Um this is about application security. You know I'm not into

regulatory compliance or OS hardening or but some of these topics will will be applicable to other areas as well. Um I'm more of an ops guy than a dev guy. I've been doing DevOps consulting for a while but my background is in operations. Um, I'm a blue more of a blue teamer than a red teamer. Uh, by far I'm like 90% blue. Um, and that'll that'll definitely be obvious from what we're talking about. Uh, if you're into application security tooling, uh, I deal mostly with ASPM stuff. So, anything before applications are put out into production. Uh, you know, SAS, SCA, stuff like that. Uh, not runtime. you know, I'm not usually looking at applications as they're executing in

production. Um, and most of this will be like an ASPM types stuff. Um, and then I do a lot personally of, you know, most of the stuff I do is cloudnative, containerized, whatever. That shouldn't be too relevant here. I mean, it will inform some of the examples, but it's not necessary to the things we're talking about. We're going to be talking more about what's in open source and you know what you do with it yourself is less relevant. Um it's just what are you consuming more than anything. Um and yeah we're mostly talking about open source here although some of this stuff you know some of the concepts will be applicable to proprietary software as

well. Uh okay so how did I get started with this? This idea kind of came to me back in uh December. there was an uh malware attack on uh Salana. Um and it kind of hit me there had been that week like a series of malware attacks. I mean like seemed like two or three a day. Um and it was just accelerating and kind of realized most enterprise software projects are probably have more risk from these kind of attacks, these malware injections into dependencies. And we'll talk about what all that means in just a second. if you don't know uh than they do from traditional vulnerabilities. So like CVES, something like log for shell um those are still

important and there's still a lot of those vulnerabilities those are actually increasing but these malware attacks are increasing even faster. Um and they're not really they don't fall under things like the CVE program. Most malware does not qualify for a CVE to be assigned to it by definition. Um which a lot of people don't realize. Um, so anyway, a lot of the CVE data is kind of bad now. The like they have a fixed amount of resources to enrich all of those advisories. There's more and more advisories. So each advisory is actually less relevant. It's just not as good. It's just a resource problem. Um, so things like that, you know, scores are bogus. It's really hard to use that.

There are some bright spots on the vulnerability front like this thing called the known exploited vulnerability list. It's like gold. That's like the highest value information there is. Uh but overall that stuff is becoming less and less useful and all of these vulner these malware attacks are going you know increasing pretty rapidly. Um okay so real quick just to set you know this this terminology that we're going to be talking about. You've seen this kind of iceberg metaphor like hey my app is at the top and all this other software that I use is at the bottom right I don't I think this is a little outdated and now I've kind of changed it a little bit at

the top is your application so all the code you're developing inhouse plus maybe your operating system depending and your what we're going to call direct dependencies and then all of the stuff underneath is this other open- source that you have no control over these transitive dependencies and if you're familiar with the difference between a direct and a transitive dependency. Um it's a relatively new thing. Uh your direct dependencies are things you declare. I need numpy, I need boto3, whatever. And transitive dependencies are the things that those dependencies need. So back when I was a kid a million years ago, you had maybe lib C from your Unix, you know, vendor and maybe you bought a really expensive networking

library from CA or something and that was pretty much it. Those were your dependencies. Now since about 2008 I think when Maven came on the scene all of a sudden you have these package managers pip uh you know go all of these things can just get the dependencies from the internet instantly and then they can say oh this package needs package X and it goes and gets that for you and you don't even know it right so all of this stuff under the surface is growing incredibly fast since then in fact there's like a Well, we'll see it in a minute. But anyway, here's kind of what we're talking about. So, like this is boto3,

which is this AWS SDK, right? It needs these other packages. These are transitive dependencies. This package here has no transitive dependencies. It's completely self-contained. This used to be kind of the the most common thing you would see. Now, this is very common, right? And some of these will have transitive dependencies as well. So, you'll have multiple levels of it. Um, okay, real quick on log for shell. U, if you remember this, uh, this kind of like got a lot of people aware of a couple of other things we're going to talk about. One, the supply chain, right? Before this, I mean, people maybe had heard that term, but didn't really realize like, oh, I'm getting all these

packages from different places and they're getting packages and now all of a sudden I have this log forj thing that I didn't even know about. Uh, and I got to do something about it, right? And this is kind of like that iceberg by the way, right? Except it's sideways. Um, if you notice that. Um, the other thing people noticed here is that there's a lot of this stuff, right? This thing is getting bigger and bigger and bigger, right? Which leads to things like this. Like this is just npm. There's like over a million packages a month now being added, new packages, which leads to the CVE problem we were talking about, right? There's more and more CVEes,

right? So the same number of manh hours are being used to enrich the data for more and more of these it becomes you know kind of unsustainable. U all right we'll come back to that in a minute. Um the other thing that came out of this was uh software bill of materials. So these are just kind of like a factual document what's in my software right it's kind of a ingredients list isn't perfect but it's not terrible. Um, are we doing on time? Okay, good. Uh, but it's basically just facts about the software, right? What's in it? You know, you want to know what you're consuming essentially, right? A lot of this stuff will be those transitive

dependencies depending on when you build a nestbomb. We'll talk about that afterwards. Um but uh if you if you have one of these like your your response to log forj is going to be like a hundred times better, right? Because you don't have to go back and rescan all of your everything in your estate. You just go to the sbombs and say, is there log forj in there? If so, what version is it? Oh, okay. It's vulnerable. I need to rebuild that. If you don't have it, you got to go and rescan everything, right? Okay. So, just uh we'll come back to that in just a bit. Let's see. Couple of things to note about these. These aren't perfect,

right? These are generated by scanners. It's an automated process. Uh they're not perfect. They have a really hard time with like binary blobs. So if they're if the dependencies aren't enumerated by the package manager for whatever ecosystem, these don't work very well. They don't work well with just a bare source code repository because if you're just looking at a code repo, they don't know how to resolve the transitive dependencies. They might know the the explicit direct ones, but not the transitive ones. They work really well with containers. I will say that. Um, okay. Now, XC. Okay, I'm flipping the funnel a little bit. So, the iceberg is going the other way now, right? We've got I I'm

not going to get into the details of the attack other than two things. One, there's this guy Jatan who did it. Nobody knows who he is. Whoops. Um, and all he did was insert a a SSH private key essentially. And the idea is I can attack all of these things if everybody starts consuming this one package, right? This compression library that most people don't really think about is used uh pretty extensively in the kernel. So all of these base distributions get it. A bunch of distributions based on them will consume it and so on and so forth. And then all of a sudden everybody's got it, right? Um the other thing is this guy you know

we don't know who he is almost certainly a state actor right this is a kind of an attack that people had theorized about this he worked on like three years on this to get maintainer access to this project and then to insert this um so wellunded extremely sophisticated um and you know it just happened uh what in this it's been a year just over a year now. Uh, okay. One other thing to note, um, this happened right at about the same time that NVD, which is the National Vulnerability Database, which is where all that enriched data for the CVS goes, was kind of collapsing. Just kind of shows the brokenness there. But for a while, I

thought it might be related, but I think it was just a coincidence. Um, the other thing to take away from XZ is all of a sudden people start wondering, okay, I've got the software supply chain. Well, who is the supplier? Well, in a lot of cases there isn't one. You do have a supply. That's a forest. I don't know if that's bright enough. Um, that's just a bunch of trees, right? So, most open source consumption is much more like this than this, right? When you go to a lumber yard, you can say, "I need a certain quality of wood. I need it cut a certain way, whatever." When you go to the forest, you can't. Right?

So, most people are doing this in their language ecosystem stuff particularly, right? There's very few like commercial suppliers of of these things that have like a where you can actually go to them and say, "Are you going to fix this or whatever." Um, okay. All right. Back to this. So, when you look at this this way, you can see all of a sudden why this kind of malware insertion is really attractive, right? you just do one thing and all of a sudden you've got all of these things compromised over here. Um in the case of XZ this again this happened kind of this f unfolded over a couple of weeks. In the case of language dependencies this

happens in a couple of hours. Uh I mean incredibly fast. Um and you know finding a zero day to exploit is a lot harder. And then you've got to figure out which of these guys are, you know, uh, vulnerable to it and so on and so forth. It's a lot more work. This is a, you know, considering the or comparatively a lot easier for most, uh, especially for unsophisticated actors, right? So there's basically two types of these that we're seeing. One is these kind of criminals that are looking for Bitcoin, wallets, keys, mal uh, ransomware, whatever, right? These are like extremely unsophisticated attacks. They're basically like, uh, I found a maintainer's, you know, account. I got

his password. I can just do whatever I want now. And they just shove in the malware and go, right? Uh, the state actors on the other hand, I don't know, stuck stuckset is probably a really dated reference. I mean, god, you got to be like ancient to remember that. Um, but anyway, that was another like long con state actor. Uh it was really impressive attack. But um these guys, we don't really know what they're up to, right? A lot of them are in the they're not necessarily in the shadows. Some of these guys are out in the open, but we don't know who they are, right? They look like normal open source contributors until they're not.

Um as far as uh tactics go, I'm going to use these four T's, types, tactics, whatever. We'll see them. Uh, a lot of people have this idea of back doors in their head as something that you plant the back door and then you wait and you go back later to get it. And that's not what happens. So like in war games, I think everybody's seen this, even people who aren't 100 years old like me. Um, you know, they're talking about I put a black back door in the system and then they go back, you know, months or years later and they can still access it. That's not what happens here. Everything is happening like super accelerated pace

now. Um that Salana attack we mentioned happened the whole thing unfolded over the course of like four hours. Like from the time the malware is inserted until the time the whole thing is wrapped up and contained and over was four hours and they still made off with you know a million dollars or whatever, right? I mean just incredibly fast. Um the XZ attack happened at an incredibly fast rate much slower than Salana but for the pace of you know when we were talking about OS distributions everything happens at like a glacial pace and they were basically as soon as they had inserted the malware I say they we don't know who Jotan is it's probably a team

of people they had a bunch of puppet accounts that were pressuring all those upstream or downstream consumers of that library to hey we just updated the thing please consume it get it into the you Debian testing get it into Fedora Rawhide and it worked like the thing was moving through. It didn't work ultimately due to some blind luck but doesn't really matter. The point is they're pushing this stuff through as fast as they can. And keep in mind as we mentioned none of these things have the meet the qualifications for a CVE to be assigned. Now, XZ did have one assigned, but um didn't really meet the the the letter of the law. Um as far as targets,

right, the type of projects they're targeting, these kind of messy projects that have bad hygiene. There's a single maintainer. He doesn't do code review. There's no branch protection in his project. um the kind of guy that if his account gets compromised, it's very likely nobody will notice for a while, a few hours at least. Um and there's ways to find these kind of projects, like look at the hygiene and kind of score them like are these projects vibrant and wellrun and stuff. We'll look at that in just a second. Um the state actors are looking for solo maintainers first of all, right? This is super important, right? If there's only one guy that they can isolate and basically especially if

he's old and overworked and wants to retire, prime target, right? Especially if they're one of these projects. Like you've all probably seen this cartoon from XKCD. I assume everybody's seen this one. That's pretty. But yeah, there's a lot of these projects that are kind of nobody's really thinks about them on a day-to-day basis, but they're holding up a lot of other stuff, right? Like XZ. Um, and there's tons of these where they have a one guy who's been maintaining it for 20 years or whatever and he wants to retire and go fishing or whatever. Um, okay. So, those are targets tools. Uh, I mentioned this thing called open SSF scorecard is a tool that you can use to kind of look

at, okay, I'm consuming these libraries. Are they any good? I want to start doing some, you know, checking on these things. uh OpenSSF scorecard is just a series of tests that you can run against those repos and say like you know are they well-maintained? Do they have a lot of different contributors? Are things getting fixed on a timely basis? Are there a you know a number of different contributors? It's not just one guy so on and so forth. Um esbombs can provide actually a map to get all that data. So most of this data is in GitHub or whoever GitHub is really good about making this available. Good espoms will have pointers to for every library you

have in your project. Here's a pointer to where I can go and get more information, right? And then I can, you know, now the bad thing is there's not a lot of tooling to automate this right now. It's very manual. Um, but it's all available through an API. Somebody could build the tooling. Uh, okay. Now let's go back to state actors real quick. How am I doing on time? Okay, good. Um, I'm currently in the middle of doing some research looking for not necessarily the next Jatan, but looking for uh bad faith actors in open source. Um, I've got these guys blocked out. I'm not going to name this project, but this is one that we're looking at right now.

Uh, I'm hoping by like Black Hat maybe we'll have evidence, but the the pattern is what we're looking for. This is a project that has four maintainers. They're all in Moscow and they all work for um a company that is currently sanctioned by the US government, has a history of privacy violations, and this project is has no other maintainers. There's some contributors from like you know the western you know uh countries or whatever but no maintainers doing any oversight over this and it is if you're running anything in Kubernetes this code is in your in your stuff I guarantee it uh it's in cubectl it's in most of the coupe base images it's in ITO it's in

helm it's in all kind of stuff uh and a lot of data flows through this particular library uh now that said I'm not naming these guys because a none of these guys have any history of anything nefarious yet, right? And there's nothing this project has never had a CVE filed against it. There's never been anything even remotely fishy about it. And the one good thing is this is in Go, which I'll get to later. Go has actually a lot of checks and stuff that makes supply chain attacks extremely difficult. Um, but this is the kind of thing we're looking for. I found other projects with a bunch of Huawei employees that basically control the whole thing, right? Huawei is being

sanctioned. It's on the do not buy list. If you're like a government contractor or whatever, you can't buy their stuff, but you can consume open sources coming out of there. Um, so stuff like that. I mean, again, this data is available and this is all self-reported, right? I mean, we're not even looking at sleeper cell type guys like Jatan. All the data is out there. Just it's impossible to get it in an automated fashion right now. All of this that we found was by basically clicking on stuff like going into the GitHub repo, looking at the clicking on the insights tab, clicking on the contributors list, clicking on each contributor and looking at what they're

doing, right? I mean, it's just no automation at all right now. Um, so of course people realize this is going on and they start thinking like, oh, we should register and vet all these open source contributors. Yeah, this is never going to happen, right? GitHub will not do it. Uh it would threaten their position as the center of the open source universe. They're not gonna they've already basically made it clear that they're not going to do anything like this. So it's just not going to happen. Nothing is going to happen at the top and top down. Everything to find these guys is going to have to be bottom up. I think um you know it's just not

going to happen, right? Uh okay. So what can we do? So you know GitHub actually in a lot of ways makes all this data available. They make it easy for people to create sbombs if they want to. They don't make it automatic. You have to click something. Um but on the other hand, they do a lot of stuff that makes it really hard for security to be increased. Anything like I there's a good interview with um npm product manager was on a podcast a couple of weeks ago. uh basically just said like we will not do anything that makes the developer experience worse. Even if it increases security 100% and makes the developer experience 1% worse, we're not

going to do it. So I was like that's really not great. Um but we there are some things you can do. So if you are consuming open source libraries, you can pin dependencies, right? You can say instead of I want numpy, I want numpy 3.2.1. Right? Now related to that, sometimes pinning your dependency is not actually pinning it. There was a attack on a GitHub action a couple of weeks ago that showed this. People who were pinning to version 4 of TJ actions actually found out that somebody could insert something over that. So the tag was not immutable. You can pin to a hash of a commit which is a gigantic nightmare uh from a maintenance point of view but

you can do it right. Um, again another thing GitHub kind of like if GitHub made those tags immutable, it would have been a lot better. Um, a couple other things to note. If you're not doing like containers or microservices or any kind of cloudnative stuff, you're way less susceptible to this stuff, right? Since since most of these attacks rely on speed, if you're not rebuilding your application 12 times a day, you're pro, you know, the chances of you consuming library X at that, you know, in that 4hour window when it's, you know, been compromised is way less, right? If you're doing quarterly releases, it's not zero, but it's not, you know, as bad. Um there's a lot of things in language

ecosystems that are being developed. Python and Pippi, they have this quarantine project feature that's kind of new. Uh looks like it might be helpful where you can kind of put boxes around what projects you trust and which ones you don't. How do you trust them? I don't know yet. Uh Go, there's a link in the notes. There's this great blog post about all they have just a huge number of features really aim to make supply chain attacks much more difficult. Uh really great work, unbelievably good stuff. And then curated repositories are always going to be safer than like npm is a dumpster fire, right? Anybody can put anything they want in npm. No questions asked. Zero developer

friction. Whereas like something like Debian is like the most highly curated, but again XZ got into Debian. It's not, you know, nothing is guaranteed. Um all right, we're almost out of time. Just a couple of things here for takeaways, right? Things to think about here. Um, why don't I have speaker notes on this one? Now I got to read it. Uh, oh, okay. So, those transitive dependencies, right? Fixing those is very difficult. It's expensive. If you're who's is anybody here a vulnerability manager, like a primary job that you do day-to-day, they'll tell you, right? I mean, if the vulnerability is in a transitive dependency and it's not like a 10.0, know I'm probably going to defer it cuz if I want to fix it,

it's going to involve like forking something and taking on a bunch of extra maintenance and who knows when the actual project will get around to it and I can rebase. It's just a gigantic nightmare, right? So, a lot of people are just not going to do stuff there. Um, and CVE NVD being broken, those scores are probably not even accurate anyway. So, I you know, uh I'm probably only going to pay attention to something like the Kev list, uh the known exploited vulnerability catalog, and maybe EPSS scores. I don't know. I'm kind of on the fence about EPSS right now. Um anyway, the number of these vulnerabilities is increasing so fast that you cannot you should not even try to patch

everything, right? I mean, this is zero CVE thing that you hear sometimes. It's a gigantic nightmare and you'll never get there. Um, okay. Malware increasing faster. It's very difficult to protect against because you again, you don't know much about those libraries, how good they are. It's very hard to get like all of that open source insight data on all of the projects you're consuming. Um, but you can scan and produce sbombs. You don't want to eat out of the dumpster. You want to know what you're consuming, right? I mean, like if you got a dog and you walk in a downtown area, you've seen your dog eat like chicken off the ground, right? Uh, don't

do that. Um, take advantage of open source insights. Again, hard to do. Currently, an expert mode thing, but hopefully the tooling will come along and make that somewhat easier. Um, basically difficult. It's impossible probably to stop these longcon state actors. They're too wellunded. There's too many of them. we have no idea who they are. Um, pin your dependencies. And then lastly, use those language and package manager security features. Especially if you're programming anything in Go, Python, Node, you're probably screwed. Uh, there's nothing you can do there. There are no security features. Um, they just give you a chainsaw with no safety basically. Uh, good luck. Um, and then keep in mind there's nothing wrong with open

source. This is just how it works. Um, it's just really more about our expectations, right? So, you could say like "Oh that iceberg, all of the problems are in those transitive dependencies. What if I just wrote all that code in house?" Well, that would be a 20 times more work and you'd probably put more vulnerabilities into it than there are now, right? Even though the quality of code developed in house has gone up, the sheer amount of it means there's was, you know, it just wouldn't, you know, it's not feasible. So, don't get too discouraged. I mean, this has been pretty dismal and and kind of uh pretty pessimistic, but overall, I'm still optimistic. Uh the tooling is coming,

but it's just not here yet. So, um thanks. And again, uh please, you know, download the slides. Um there's too many slides here for me to get back to the beginning. Um I'm going to do it anyway. There it is. If you want to get the slides, get them, please. There's a ton more elaboration on those takeaways and stuff in the slides. There's like a bunch of like a separate slide for each one of them with a bunch of, you know, pointers and stuff. Um, feedback welcome. Uh, I know that was a lot of information in like 28 minutes, but I do appreciate everybody's attention. Thanks a lot. Oh, and if I'll be in the hallway

for a little bit and I'm going to go to the circuit. I said that earlier. I'm going to go do the circuit building thing.