Are the Bad Guys Already in Your Software Supply Chain? (Spoiler Alert: Yes)

Hey everybody. Um, get my speaker notes up on here. Uh, am I in the way? Not really. Okay. Um, thanks for coming. My name is Paul Noverise. Uh, I work for a company called Hunted Labs. We do software supply chain security consulting services etc. Um, this QR code, I know everybody likes to go to a security conference and get blasted in the face with QR codes. Um, but that does go to my GitHub repo where these slides are. And the slides in the repo have the speaker notes and stuff. So, they're actually better than the the slides you're going to see. Um, you you know, if you have questions, you can email me, hit me up on Messedon

or Signal, whatever. Um the agenda is pretty short. I mean this is not the real agenda, but basically there's a lot of bad things going on and at the very end it's really a lot of pessimistic stuff. It's pretty dismal and then at the end we'll talk about some things that we can probably improve some things somewhat. Uh just some biases. Um you know basically this is about application security. I'm not going to talk about like regulatory compliance or like OS hardening. Some of the things we'll talk about do have relevance around operating systems as well. There is a operating system supply chain very similar to application you know uh open source supply chain. Um I've got a big ops infrastructure

background. I've been doing DevOps for a while now, but my background before that was purely infrastructure. Uh I'm much more of a blue teamer than a redteamer. Um for those that are not in the appseac universe, um basically my job is very ASPM ccentric. So that is application security posture management which is Latin for everything you do before you deploy the application. Right? So, as you're examining code, things you do before deployment. Um, kind of like if you were in the keynote this morning, right? Using the eraser rather than the sledgehammer, uh, SCA being software composition analysis, which is basically looking at dependencies, the short version of this. It was originally developed for things like licensing compliance, but it's very

good for these kinds of uh, exercises. Um, just as a biased note, I basically live inside containers. Uh, this talk is not specific to containers, but I'll use a lot of occasionally fall back on that. Um, but all of the concepts are applicable to anything whether you're containerized or not. And then I've been in open source for I'm not going to name a number, but a long time. Um, some of these concepts do apply to proprietary software as well. I mean, there's a lot of proprietary software that has open source components. Uh, but most of this is very open source specific. Um, okay. I'm going to start with uh just a quick recap of both log for

shell and xz. I mean, these are two kind of different uh situations, but they have a lot they both have a lot to say to inform all of these other things we'll be talking about today. Uh so this is kind of like the map of a log for shell and this is kind of a lot of people it was the first time they really they may have heard of software supply chain but this was the first time they kind of understood it. It became tangible because for a lot of people log forj was not a direct dependency but a transitive one and they didn't a lot of people didn't even know they had it. They

weren't sure if they had it. This also became an exercise where a lot of people became aware of software bill of materials um which has come up a couple of times today in some of these sessions. Um we'll talk about sbombs a little bit later. Uh but after log for shell right it was a a lot of upheaval and a lot of activity in a short period of time. It felt like a long period of time, but when it was over, everybody kind of went back to what they were doing before, right? An advisory was issued, we remediated it, and that's it. We're back to normal operations. Nobody really changed anything about how they're consuming any of this stuff. Um,

and there really wasn't any lesson to learn about using open source other than sometimes these things happen, right? Zero days happen. Uh, okay. Then after that, before XC, a bunch of other stuff happened. So all the nobody's I wouldn't say nobody, a lot of people's behavior did not change, but the situation out in the wild changed considerably. Um, we've got, you know, the NVD sort of starting to collapse under its own weight around this period. Um, we'll talk about the growth here. Like this graph on the top right is the number of advisories issued versus the number of advisories that get enriched with data by the national vulnerability database. All of a sudden that kind of

came to a crashing halt. Um and around the same time we had XZ occur. Plus, XZ was just one of a ton of supply chain attacks where basically people hijack a repository somehow, either a maintainer account or becoming a maintainer in this case, and then inserting a malware payload, right? And this kind of well, we'll talk about some of the brokenness here in just a minute. There's a whole separate slide, but just really quick, this is kind of the opposite of that previous funnel where now instead of looking upstream, we're looking downstream. Once XZ gets compromised, it spreads through all of these other things, right? As these things consume it, right? SUSA, whoops, Fedora, I was

trying to use the laser here. Uh, Debian, they all absorb XZ as a dependency. We're not going to get into the particulars of the attack because that doesn't matter. But then all these other things downstream of those uh start consuming it. It didn't really get this far. It really only got this far, right? But this was the idea was I can compromise here and then get all this stuff. Um let's see how is there a clock? There's no clock. I just got to watch it. Okay, that's fine. Oh, I got one right here, too. Okay. Uh All right. So now we'll talk about vault management being broken. So uh this is a graph of just the number of

packages in not packages but the number of releases in npm uh over time. npm was introduced in like 2010. Um this is from a talk of ex-corker of mine gave at a few months ago. It's called open source is bigger than you can imagine. I highly recommend it. There's a link to it in the footnotes. Um, but it's mostly about how big open source has gotten without people really noticing, right? And so these these are just releases in npm and it's just going up. I mean, like 8,000 new releases a day, I think is the latest I saw. Um, so there's really no way anybody can keep up with this, right? And and at the

same time, I mean, obviously CVEes are growing at the at a similar rate, right? And you'll notice right around here is where things really kind of well right in this area. Package managers for language ecosystems were kind of introduced like around 2010. NPM was around introduced in 2010. I think Maven a couple years before that but in the same area you got PIP. All of those things happening about the same time right? Um, so you end up with this huge increase in CVEEs and the National Vulnerability Database, their budget is actually going down at the same time. So it's kind of inevitable that those advisories, the enrichment, the data enrichment of them is becoming worse and worse over time.

There's just less and less uh resources, fewer and fewer resources to be shared by more and more advisories. Right. And most of well not all of this is open source but if you take out just the open source right the graph is probably a little steeper actually. Um okay so clearly I mean this is kind of unsustainable right as we'll see in just a minute. Uh okay. So, oh one other thing I wanted to mention here. Let me go back. CVEes, right? The common vulnerability. What is it? And then I don't know what CV even stands for anymore. My brain is so fried. But um it's basically this this database of these advisories, right? This whole

system was developed in the 1900s, right? I mean, open source was a thing then, but not anywhere near what it is now. Um, these tools just haven't really been able to adapt. They've just because of that resource starvation, there's just not a lot of adaptation going on here. All right, so supply chains. Um, this is kind of like the old days, right? I don't know. I guess everybody's seen this movie, War Games, I'm assuming. Um, the reason I have this though, this computer, those switches on the front are for basically inputting one bite at a time, right? And then I mean it does have disc drives and stuff, but you had to basically get this thing

bootstrapped one bite at a time. So in that universe, right, dependencies are not a thing. If everybody I guess maybe I shouldn't assume everybody knows what a dependency is. A dependency is just somebody else's code that you're dependent on, right? you use their code and then write your application on top of it. In this universe, those don't really exist, right? Even after this, you might have like lib C. If you have a commercial Unix system, you might pay CA a couple million bucks for a networking library or something like that, but for the most part, that's it, right? And then you start having languages like Python and Ruby and Node where it's much easier to develop libraries and

distribute them more importantly. And once you have these package managers that enable you to resolve dependencies automatically and just install things, you don't have to go out to the internet and find NumPy and then make sure you put it in the right place and you got the right version and you have to do all that manually in the old days. and now it just happens and it all works and you stop paying attention, right? Um, so this is kind of a different universe. So that takes us to this metaphor where it used to be I've got this iceberg of software and my application is on top and everybody else's code is on the bottom. Uh, and

then in this old version, you know, this used to be the operating system and libby C or whatever, but I've kind of changed it a little bit and now the top is your application things you control or decide on your direct dependencies. I need botocore or whatever. I need Debian and those I make decisions on. And even if if you're a containerized, you're probably not even thinking about the operating system. That goes down into the bottom almost. Everything down here are these transitive dependencies. The dependencies of your dependencies. I installed botocore for Python and I get a bunch of other libraries installed and I don't know what they are. I might be paying attention, but they're scrolling

by on the screen at, you know, 3,000 miles an hour. Um, now to an attacker, it doesn't matter. All of this code has vulnerabilities in it. All of this code can be attacked one way or the other, whether it's through a vulnerability or through a supply chain attack. We'll talk about the difference in just a second here. Um, they don't care. But you can only really work here. When you try to fix or repair things down here, it becomes extremely tedious, expensive. It's actually almost impossible in a lot of cases. you have to wait for and hope that whatever your direct dependency is rebases their dependency before you can do anything here or you have to do

something crazy like fork it and then maintain it yourself which is even worse. Uh so generally what happens is who's is anybody here in vulnerability management like as a primary thing do you spend a lot of time like oh it's a transitor dependency I'm going to put it on a list of things to ignore. Well, a lot of people do. So, congratulations for not doing that. But that's what generally happens is a lot of people just say, "We're gonna, unless the thing is literally on fire, we're going to ignore it for now and maybe come back to it later." It's really not great. Um okay. The other thing that happens here is now that you're using these package managers

and you're not getting libraries from a commercial software vendor like Sun or Digital and you're getting them from npm, you've gone from going to the lumber yard to going to the forest and chopping down your own tree. So if you go to the forest and you chop down a tree, you can chop down as many trees as you want. If the tree is rotten and the board comes out warped or whatever, you can't really go back to the forest and yell at it. I mean, you can, but nothing will happen, right? And people will think you're crazy. Uh whereas, if you're buying wood from a lumber yard, you have expectations of the quality. You can go

back to them and get, you know, some a replacement. Um, so things like Red Hat or, you know, those are vendors. NPM is not a vendor. All right, let's see. All right, now to the good stuff, the supply chain attacks, right? So what I mean, what is differentiating XZ from log for shell? The biggest thing is like this is on purpose, right? There is a person here inserting some kind of payload whether it's malware, ransomware or whatever into this and how they get access to it is kind of a different like a vector thing but whereas log for shell was just a what we would call a vulnerability a kind of inadvertent a mistake essentially and this is on purpose right

that's the biggest difference here um and if you go by the Look, if you go to CVE and look at their definition of vulnerabilities, this doesn't qualify. This is software doing what the author of the software intended it to do. It's not what I intended, but that's because I didn't know what I was getting, but this guy intended this thing to do this. So, it's not a vulnerability. you know, XZ did get a CVE number assigned just because there was so much, you know, people were basically going out, you know, with pitchforks and torches. Um, but most of these attacks don't get CVE numbers assigned. So even if you are going into you're you're doing all your

scanning, you're matching, you're getting all these CVE matches and you get every one of them, you're down to zero, you're still going to miss most of these things here, right? You're not going to be aware of them if all you're looking at is those CVE numbers. So just more of that uh brokenness. Where was I here? Sorry, I was had my speaker notes on the other slide, so it's a little little disjointed here. Uh, okay. So, now we'll talk about these tactics that the supply chain attackers are using, right? So, back to war games. These guys are talking about back doors, right? And in this scenario, like as much as this movie informs people about cyber security in a

lot of ways, and it's still very relevant. This particular discussion here where they're talking about backdoors is completely wrong, right? It might have been correct at the time, but things have changed so much. Right? In this scenario, they're talking about backd doors as something you put into a system and leave and come back to months or years later, right? It is not what happens now at all. What happens now is things move super fast. Right? Once the code is delivered, whether it's uploaded to npm, whatever it is, it's pushed, everything accelerates, right? Because the clock is ticking. They know it's going to get discovered at some point. So, they're trying to get as much whatever as fast as they can. And that

whatever is sometimes, you know, Bitcoin, they're trying to get your wallet. they're trying to get your keys, whatever it is, access to something. Um, and in the meantime, a lot of defenders are just not even paying attention to this. They're just consuming all of that open source the same way regardless. And so, these guys just go around the defenses. Um, they're just not prepared for these kind of attacks, right? I mean, even though they're becoming more and more common, they've really accelerated just in the last couple of years. So what do we do about this? Right? So my company has started some research. I'm doing a little bit of digging around in open source and trying to find like

how can we identify these things before they get blown up. Uh okay. So here was the idea was this is what uh I think this was back in December when I thought about this. I was like CVES are going up. There's more and more of them every year, but the relevance of them is going down. The actionability of them is going down. And actually, all of these supply chain attacks with malware insertions are increasing even faster. Right? That was kind of my my theory. It's like, well, how do we find this stuff? Right? How do I look for it? How can I test this theory? Uh oh my god, I forgot to put this one in. If you remember the scene

in Seinfeld, the the voice, right? In order to manage risk, we must first understand risk, right? Well, we got to figure out what these how these supply chain attacks work in order to try to catch them before they blow up, right? First thing we notice is basically two types here. We kind of mentioned the Bitcoin thief, right? The the the small time criminal. These are like super simple attacks, right? They're just I'm gonna hijack somebody's account. I'm gonna fish his password, whatever, and then steal a bunch of Bitcoin, right? Like Salana, uh, Salana Web3.js was back in December. The whole thing, oh, I thought that was a question. Um, the whole thing happened in like four

hours, I think, from the time the code was, you know, committed to the time it was all wrapped up and they made off with, I don't know how much money, but a lot. Um, then you have these kind of like weird state actors that we don't really know much about, right? Some of these guys are these are the long con guys like Jatan that did XZ, right? That was a three-year operation, right? this guy. It wasn't one guy. It was almost certainly a team of people using one persona that spent three years diligently working on an open source project, doing just normal code commits, maintenance, gardening, all the good stuff that they're supposed to do. And

then one day, boom, it's all over, right? Everything happened there. And once the code, again, once the code is committed, everything accelerates. And then there's all of a sudden a network of bot accounts, puppet accounts pressuring all of those upstream projects to to accept the new version of XZ. And the whole thing happens in a couple of days, right? Which, you know, I'm talking about a couple of hours over here. A couple of days on in in a in an OS level package. That's like light speed right? Um, so the targets, right? There's kind of two different things. the these guys are looking for and these are kind of similar for both the criminal and the

state actor one is this kind of messy desk thing right these projects that are amateur I don't want to drag them I mean there's a lot of very useful software out there but the maintainers are not necessarily super diligent about best practices right maybe they don't have branch protection they don't use two-factor authentication on their GitHub account they're not fuzzing they're not whatever right but these kind of projects when you you can find this stuff. We'll talk about this thing called the OpenSSF scorecard in a minute, but those kind of projects tend to have lacks security policies if they even have one, right? There there's more opportunity to get in there essentially, right? Um, and the

second kind of target here are these solo maintainers we talk about a lot. We we'll hear this a lot and there are a lot of these projects. I mean a lot if you go to that uh open source is bigger than you can imagine talk I mean the number of single maintainer projects in any ecosystem you pick it doesn't matter if you go to node you go to python go to go whatever all of the graphs look the same and there's a huge spike on one end of single maintainers and then the two maintainers is like way down here and then it's this long tail and it doesn't matter they even excluded like I only want to look

at projects with more than a hundred,000 downloads and the graph looks the same. Make it 10 million downloads, the graph looks the same. Right? So, there are tons of projects where there's a single maintainer and if this account gets compromised, there's nobody else to that will notice what's going on essentially. Um, and then there's this kind of set of like really critical projects like XZ that most people don't think about on a regular basis, but they're so connected that if you can compromise them, you can kind of do outsides damage. So this intersection of these three things is like really critical. All right. Oh, I put the graph in. This is the graph, right? I mean,

this is one maintainer and then you can see it just like drops like a, you know, off a cliff. Um, okay. So, we can talk about tools like what can we do about this? Right? As I'm consuming all of this open source, there's all these weirdos out there. They're doing all kind of weird stuff. How can I protect myself? Right? I don't want to uh, you know, I'm going out to npm and just I'm literally eating out of a dumpster, right? I'm just grabbing whatever and consuming it. So, there's the scorecard. Whoops. Go back. This is this OpenSSF scorecard. It's just a way of basically evaluating open source repos and giving them points for doing good

things. Um, there's a lot of ways to automate this. Uh, it's really straightforward and it's pretty simple. Uh, and it can give you a real quick idea of the health of a project. Uh the second thing is these the pizza rolls here is kind of the the standin for the sbomb, the software bill of materials, right? And if you don't know what that is, it's essentially an ingredients list and nutrition facts, whatever you want to call it. It's what's in the software that you're consuming. So this JSON underneath here is kind of an example. And the thing at the very bottom is the real key. A lot of software security tools use an SBOM to do things like

match vulnerabilities. They'll there's no CPEs in the on the screen here, but there are CPE. Oh, no. There's a Pearl. So, they can match vulnerabilities with a Pearl. There's some license data in there. I don't think that's on the screen, right? And that's pretty much all most tools do with that. But if you have this information, where is the source of truth for this project, this library, whatever it is I'm consuming? I can then go to GitHub or wherever the project lives. And GitHub makes a ton of data available in their insights. And this is way more than just what's in the software. It's who is contributing to it, what are how frequently are they doing it, what how

quickly do things get fixed, you know, stuff like that, right? all of that stuff we were talking the open SSF scorecard gets a lot of data through here but there's more data in here now the problem is there's not a lot of tooling to do this at scale right now right a lot of this is people will just start clicking on this and just seeing what they find right uh who's you know does this project have any kind of standards do they have a security policy whatever but it's hard to do this at scale I mean this is just kind of like some of the data you can find like, you know, how often are people contributing,

blah, blah, blah. Um, there's a ton of it in there. And like I said, it's hard to manage it, though. There's no real way to scale a evaluation of a large application that has thousands of libraries, right? I mean, you're not going to sit here and click on a thousand different GitHub repos and then click on contributors a thousand times or whatever, right? There's just no way to do that right now. Um, there's a couple of like projects like kind of eating around the edges of this, but it's at a very small scale right now. Uh, and most of the ones I've seen are only doing things like I flagging repos that have a single

maintainer, right? Which is important, but it's that's, you know, it's kind of early stages. Right now, what we've been doing research- wise, like here's the, well, the idea here is I want to know, am I eating something that's cake or is it toilet paper? Right? I mean, um, so we talk about the, you know, the XZ problem, Jatan, we can look at these contri, you know, contributor data in GitHub. The next thing I hear is, "Oh, well, why don't we why doesn't GitHub just make open source contributors submit a driver's license or some kind of register and vet them?" Like, okay, who is going to do that? GitHub is not going to do this, right? First of all,

GitHub is committed to maintaining developer experience. They're not going to do anything to increase the friction level for contributors, right? First of all, because if they did, a lot of people would leave and they're not going to threaten their position at the center of the open source universe. And then second of all, who nobody would actually do all this vetting. Who's going to pay for that? Nobody, right? Um, so it's just that's not going to happen, right? This verified open source contributor. I see this about every few months somebody's like, "Yeah, we should start, you know, papers, please." Not going to happen. But what we've started to do now this is these are actual contributors. I'm not going

to out this project. But I've we started looking at stuff as I mentioned in my company. I've got a threat hunter on staff and I've been helping as well and we found a couple of projects where this is one. It's a go library. It's in a lot of stuff and the maintainers all there's four maintainers. They're all Russian and they all work for uh a sanctioned organization or a subsidiary of it. And there's there are maybe one or two Westerner contributors here, but they're very minor contributors. So, there's basically no oversight outside of this sanctioned organization um that has been involved in other stuff. Now, these I've got these guys blurred out because I'm not outing them

in particular, right? There's nothing in this repo that is looks to be malicious, but this is the pattern we're looking for, right? So, some of these guys are like undercover like Jatan. That's not a real person. These guys are real. We can we found their their offline, you know, personalities and stuff. They're real people. Um, but we don't know. I mean there if you were the FSB and you were looking to compromise somebody, you could lean on these guys pretty easily probably, right? Uh we found similar things. Again, I'm not singling these guys out. It's just a an indicative of a pattern. We found similar projects uh with Chinese maintainers. Uh and when I say

China, I mean mainland China. Uh didn't find anything North Korean. uh which like I don't think there's anyone on the internet who is openly I'm from North Korea. I I just can't find them. Um but they're out there, right? We know they're out there. Uh and then we did find out quite a bit from Iran, too. Uh and again, none of these are they're all kind of like in these weird packs. Like these guys contribute to the same projects. Like they're all working together, right? It it's extremely fishy. I we've got a report coming out. I'm hoping to have something at Black Hat that will actually have some actual evidence. Uh maybe we'll unmask these

guys if we find, you know, some more stuff. Uh we'll

see. Okay. Um yeah. So things are bad, right? I mean it's not good. the the numbers are are bad. There's just so much going on. Uh the tooling is not there yet, right? There's, like I said, there's some, you know, we're working on something, whatever. Every a lot of people are working on tooling, but it's all embriionic right now. Um the bad guys are winning. I like you just read the news. I mean, there's another malware uh compromise every day basically at this point. Um, you know, and like I said, GitHub has the ability, they're positioned to help, but they won't, right? In some ways, I understand it. I get it. Like, they don't want to increase developer

friction, but they also don't want to increase developer friction. Right? There was there's a link in the notes to a interview with an npm product manager that I just heard about two or three weeks ago where he basically said like yeah we evaluated a bunch of security features and we decided to do none of them because it would make a developer's lot it would increase the time for for a developer to make their first push essentially we want people to push code as fast as possible right I'm like well that's kind of part of the problem. Like I like I get it. I mean I'm very sympathetic to that mindset, but the problem I mean it's really bad. Um so I

do have some kind of a few takeaways here that are things that we can do. Um one is we talked about dependencies a lot, right? And what happens in these attacks is a new version gets released, you consume it because you're just grabbing whatever's out there and you get compromised. So you can do something called pinning your dependencies. It's kind of a controversial take where you're basically saying I only I want library X version 1.2.3 or actually that's even not really accurate. As it turns out, like there was a GitHub attack a couple of weeks ago on a GitHub action. If you pin to a version, that's actually not immutable. An attacker switched out what was in version

1.2.3 and so everybody consumed it even if they were pinning to that version. So you have to pin to an actual hash, a commit hash, which is a lot more difficult. Um, from a maintenance point of view, it's it's not clear what you're pinning to. You just see a random string of characters rather than a version number. So it creates a lot of maintenance issues there where you have to document what you're doing and stuff like that, right? But it is possible to essentially freeze in place. So if malware does get introduced, you're not con just blindly consuming it, right? It causes other problems. If a fix is released, then you have to go out and

ask for it specifically rather than just get it automatically, right? So it's there's tradeoffs. Um I used to think that pinning was bad and now I'm like it's pretty good. Um, we talked about GitHub. One other thing, if you're not like very DevOpsy, this is much less of a concern, right? If you're building your project 12 times a day, you're much more likely to consume something in that short window when it's compromised, right? Because these attacks move so fast now and they get resolved relatively quickly. If you're only building, you know, you're releasing once a quarter, this is probably less of a concern. concern. I mean, it's not a zero concern, especially if you have

thousands and thousands of dependencies out there, right? Eventually, you're going to get something, but it's not like an existential thing that you should be losing sleep about every night. Um, a lot of the repositories like Python's Pippi are introducing features to combat some of this stuff. Pippi has a package quarantine thing that's kind of in beta, I think, where you can kind of put things in a box if you don't trust them and kind of only consume them on certain conditions. I don't know. I haven't really had time to play with it that much. It's pretty new. There's links in them in the notes. Um, Go, if you're using Go, Go has a ton of

features to really make supply chain attacks a lot harder and slower. That's another reason that that that repo I kind of alluded to earlier, it's a Go repository. So, even if they did something malicious, it the it might be contained, right? I don't know. It just depends. Um, in general, C highly curated repositories are way safer than things like npm. npm is, you know, the wild west. It's total anarchy. Anybody can put anything they want in there, right? You can put the dead bodies there. Whereas something like Debian is like extremely highly curated. But on the other hand, if you look at XZ, that did get into Fedora's rawhide. It got into Debian testing, right? Those are

some of the most curated repositories out there and it still got in. um CVE, NVDs, it's all broken. I don't have a lot of, you know, hope that that's going to get fixed. All of this stuff kind of exists outside of that universe anyway. So if you are trying to protect things right the the marginal value of your time every minute you spend working on vulnerabilities you're spending more and more effort to do less and less remediation basically it takes more time and the fix you do it has less value essentially so you you have to basically do 10 times as much work you know it's just getting harder and harder I mean that's nobody in no one person's fault

but it's just like I said a kind of a confluence of resource constraints and the extreme growth in those libraries, right? I mean the the ease of installing those libraries has contributed so much to the expansion of them that it just these things kind of grow out of control and nobody can stay in front of it. Um I do recommend Sbombs pretty highly. like scanning everything you have generating sbombs even if you're not doing anything with them today just having them is better than not having them if something occurs you will know you don't even need an advisory to find out if you're affected right as soon as you hear about log forj or whatever the next zero day

is or the next zexz you can find out if you have it and you're affected you know without having to rescan your entire estate uh it's just really managing sbomb is a big problem. You know, they're not a silver bullet or anything, but you're better off with them than without them. Don't eat out of dumpsters, right? I'm going to use that analogy all the time, right? My dog, if you walk your dog in the city, right? Your dog has eaten like chicken off the sidewalk. I'm sure a lot of you have had that experience. Ice cream cones, pizza, whatever. Like, where did that come from? Like, they don't care, right? They just eat it. Um, yeah. And this, you know, not to

harp on CVES. I mean, but you can't the other end of it, right? I mean, one thing, the advisories are less and less valuable, but on the other end, there's so many things now, even if you spent all your time patching things, you can't keep up. There's just no way. There's it's growing faster than you can patch, right? So, it something else has to come up. I mean, I just don't have the answer, right? Like I said, some of these things are just really bad right now. Um, and then the last thing, these state actors, I mean, it's a big problem. We don't know how many of them are out there. They're really wellunded. They're extremely

tenacious. Uh, they don't care how long it takes and it's going to be really hard to find them. I mean, some of them are mixed in with normal open source developers, right? There's a lot of people who get recruited out of college in China or whatever and they're working for they're not embedded in the PLA or whatever. They're at Huawei or whatever company and they look like just a normal they are a normal open source contributor until somebody flips the switch. Um, you know, I mean that's extremely dicey, right? It's really hard to to to guard against that kind of stuff. Uh and then the last thing you know find out what you know whatever language

you're using whatever you're writing your applications in find out what the the security features are of that language that ecosystem and use them right I mean they're they are putting in the work you know there's a lot of stuff coming out that makes these things a lot harder for the attackers even if it's not perfect any speed bump is better than nothing right um again the QR code. Uh, we've got time for questions. Uh, that went a little better than I thought it would actually. I was kind of worried about the time, but um, yeah, if there's any questions. Yeah, go ahead.

I did want to point out that Oh, yeah. No, no, thanks. Okay, so he's bringing up like that that slide of the four guys, right? Yeah. Whoops. Yeah, this data is actually just self-reported on GitHub. Right now, there's a couple of things. We've actually I've my company has a database going back about 20 years of this stuff. And we have noticed a lot of people changing this stuff like removing like see not all of these guys have their company on here anymore. And they all did at one point. Um, another thing is though we're we're well there's a lot of assent type techniques you can use to try to tie a GitHub account to an offline

personality, right? Well, I can set my personal profile right now. Yeah. Oh, yeah. Labs organization. Yes, you could. You can't stop, right? I can't stop you, right? But you can figure out like I mean there are thing there are there's some companies that actually do this work like that will give you feeds like I can type in GitHub account XYZ and they'll give you a phone number or they'll tell you they'll say where this guy went to school who he's gotten grants from what governments he's you know things like that. Um but they'll also tell you we have no information on this person or if you did that they would say we're pretty sure this guy

does not work there or whatever. Right? So there but those are expensive like those are really expensive services like I'm sure Palunteer has something like that. I've talked to a couple of other companies and I mean we're kind of looking at leveraging something like that to you know do some other work. I can tell you I have absolutely seen Huawei employees removing Huawei from their GitHub profile and I don't know why they're doing that. It's not just one or two of them and they still work there. pretty sure like that kind of stuff like makes me really suspicious, you know? I not that I'm a you know 007 or whatever, but that kind of makes my

hair stand up a little bit like what are they up to when you see stuff like that. So it's not just like what's happening now that's important. It is having the ability to look back and say what's changed. So a lot of that if you look at like the GitHub insight stuff wherever that was I don't know this stuff looking at what is going on at this exact moment in time is kind of illuminating but looking at how things are are unfolding over an extended period of time is much more illuminating like okay there's this kind of rhythm to this project all of a sudden there's this huge spike in activity what's going on there right those kind of things so

yeah the fact Yeah, it's very important to keep in mind like you said though that data on these individual contributors is self-reported, right? And they can absolutely put whatever they want, right? So the Jatan persona was made up, right? There's no such person, right? Anybody can make a GitHub account and generate an AI. You know, if I see an anime avatar, I'm like, "Oh, I'm suspicious already." I don't know. I'm not really, but I'm I kind of making jokingly made that, you know, when we were looking through a profile and there were a couple of, you know, Chinese contributors and they both had anime avatars. I was like, "Oh, yeah, definitely. Those guys are definitely up to no good." Um, yeah,

good good point though. Yeah, absolutely. Uh, somebody else over here had a Yeah, go ahead. One of the things that we see is that people will publish a new version of their package um and they'll say, "Oh, here here are the commits that went into it." And some of those commits might have security impact. There's no CVE assigned because like you said the CVE program is not necessarily complete. But you know our execs whoever will look at that and say like oh crap we need to start taking these updates as quickly as possible right now. We have tools like dependabot in Nougat or in in GitHub um that will update your packages for you. And the

cadence that we're being asked to do this in, I've seen repo owners automatically start merging dependabot commits. Yeah. Uh dependable PRs. And now if you're saying like, hey, there's a good opportunity for people to inject malware into packages like where how does one reason about where the line should be drawn here behind move fast and don't move so fast that people attack you? Yeah, that's a million-dollar question. I mean like I said like if you asked me this like right after log forj right log for shell specifically I would have said you're nuts to pin dependencies you should consume every update as fast as you can because you're going to get all those fixes right but now we're seeing

more of the the the amount of like supply chain malware attacks at that point was not anywhere close to what it is now right and so the risk level goes way up but you are skipping a bunch of security fixes unless you actually know about them, right? Because there's tons of security fixes that fix things that aren't CVEes that have no number assigned and the the developer, the maintainer knows, ah, I fixed a security problem, but nobody else knows unless they're looking at that change log and they read, oh, buffer overflow, whatever it is, right? I mean, buffer overflows or not, but yeah. Yeah. I mean, I don't know what to tell people. Like it

is like on the one hand you should be consuming updates. You should update early and often but on the other hand you can't just like blindly eat out of the dumpster. But on the other hand if you don't eat out of the dumpster you're going to miss some good stuff. I don't know. It's if you had you know if you had the ability to know exactly what was in the dumpster and sort out the good from the bad, right? then the problem would be solved and you'd probably make a lot of money right? Hopefully, you know, I mean, a lot of people would pay for that, right? I would watch other people eat out of the

dumpster and get Yeah. Yeah. Well, that Okay, so that actually is not a terrible idea, right? So, like that that Python quar package quarantine, right? I don't know exact it doesn't exactly work like this but in my mind the way it would work was you could set a timer or something and say I'm gonna I want the latest updates except I don't want anything from the last 12 hours or whatever, right? And then 24 hours, right? I mean that's kind of hard. I've seen people try to set up like a private repository to mirror updates and they set a time delay on how often they sync it or whatever. I mean that's a lot of

maintenance work though. you can't. Most most shops don't have the capability to to take on that kind of work. That's a lot of overhead. So, there's not a good answer to that. I mean, you're you're in a bad spot no matter what you do. Again, like a lot of this stuff is bad news, right? I mean, there's there's the promise of these things getting fixed at some point, but there it's not today. Yeah. Anybody else? I think we're at it. Yeah. Go ahead. I think we can do one more. I'm curious as to why you think GitHub won't do Oh, they've said it. They've absolutely said it. Okay. He's as the question was why why I think

GitHub won't, you know, do the things to improve this stuff. They've done some things like they have made it easier to create sbombs. Like you can just one-touch sbombs get created on every push or whatever, right? I mean, they've done some things, but things that will make developers angry or slow or whatever they won't do. and they've been completely upfront about that like we will not do these things. Um, like I said, there's there was an interview just a couple of days ago. I mean, that was with an npm product manager, but they're all kind of mixed together at this point, right? Since Microsoft bought both of them. Um, but there's other I think there's a

couple of links in there to to actual GitHub security teams and stuff like that just saying like the developer experience is the number one priority. Period. The end. security comes, you know, later. You know, I I get why they made that decision. I don't necessarily agree with it, but I they do have some logic behind it, but I mean, if you think about it in the big picture, despite all this bad stuff, right, most stuff works still, you know, it's not a total apocalypse. So, maybe they're right. I don't know. But yeah, you're I mean, that's a good question. Okay, we got two minutes, so we can do one more question. Maybe I have kind of long-winded answers, but

these are good questions. No, no. All right, we'll wrap it up. Thank you so much.