It's the middle of the night; do you know where your maintainers are

All right, apologies starting a bit late, but I hope I hand on time. So, hello friends. This is my first time presenting at Bsides, and it's with great pleasure that I stand before you on this reasonably sized stage to talk to you about some challenges that we face in open source. First, a small introduction for those of you who don't know me. That's me. I'm the speaker. It's me. So I'm an engineering manager at Canonicle and in my day-to-day I coordinate security maintenance for 15 years just announced yesterday for all Ubuntu LTS releases. So I'm maintaining packages from 20134 until the present. And whenever I have some spare time I also like to do board

gaming and yeah trying to boil the never- ending ever growing vulnerability ocean one spoonful at a time. As for the middle of the night while I was up making these slides I do not speak with my slides. Those who speak with their slides have forgotten the face of their elders. I speak with my heart and I left the slides to the last minute because I believe in the heart of the cards. So here we go now to set up some ground rules. Gr sorry sorry sorry about that. U some quick ground rules uh so that we understand each other during this presentation. All the opinions expressed are exclusively my own and do not necessarily reflect the views of my

employer. They do however reflect my experience working there. So, as I said, I do vulnerability management for a dishro. So, bear with me. We're going to talk a bit about it. And yeah, my slides are ugly, but they're always made with love. Anecdotes are not data, but do they do reflect my experience and data should be accurate except when using hyperbole for flare. So, please keep your arms and legs inside the vehicle at all times, but I will be requiring your arms in a bit. And let's go. So, what is the status quo? Show of hands. How many of you have used open source software? I'm going to pretend I'm seeing because I can't see anything. There's a light

over there shining in my eyes and I wear glasses. But this is what I expect the audience to be like. Everybody just put your hands up. If you didn't raise your hand, you're probably just not aware of what I'm saying. I see you there in the laptops and scrolling on your phone or well just lying for the lols. That's quite sus of you, but alas, let's keep going. It is a security conference after all. So open source code is in pretty much everything. The numbers vary between studies and methodologies on how you're determining if something uses open source, but it's generally over 90%. You'd be hardressed to find a study that says that open source usage is not at

90%. For all practical purposes, there's some open source software in every better supply chain. For those that have been to the US or live there, it's the high fructose corn syrup of software. It's cheap. It's plentiful. you put it in everything. So measuring the ecosystems uh is also a very difficult target like packaging ecosystems are in constant growth. These numbers are for from 2024 and the exact second that this slide was created they were already out of date. Ecosystems like npm for which I have great disdain please uh but I must give it credit are pushing the the growth of packages to incredible numbers every single day. There's thousands of packages being created and the ecosystem just continues

to grow. The horde must grow. GitHub as well is a is a good indicator of this. GitHub hit 100 million users even far before. They predicted they would hit 100 million users in 2025. They hit 100 million users in 2023. You still go on hit go github and you find a lot of repos with one or two commits or the same copy pasted template apps but it's a show that people are joining in and they want to contribute and they want to share code. On the latest Octoverse report they described that a new developer joins GitHub every second. So in the time that I've been on stage their numbers have once again grown. By the time I finish the talk

we'll have a bunch more other developers there. And AI leads TypeScript into being the number one language. OPM. Oh, I hate the but is what it is. So how do you think how do I think about open source? And to that to that I ask you what is it that you truly desire? Let's say that you want a train running across your terminal every time you mistype lssl. Well, all aboard there's probably a package for that. There goes a train. I'm not going to skip this slide. I want to see the train get there. It is very important to me that the train gets there. All right, we got there. So cool. Yes, indeed. Open source is so cool indeed.

But it's not without it challenges. Like many of you are probably looking at that thing. Uh but why does a package exist that does that? And that is the core question that creates some challenges regarding open source sustainability. I believe the ease of access to it is a double-edged sword. It's great. It's ubiquitous. There's something for for everything that you might need, but also creates a challenge given the way that humans work and think about things. Jen's wonderful presentation yesterday talked about the hacking scene. Who here remembers the other scene? You know, the the waters one. Remember those terms over there? Were you a courier, a trade, or did you just go, "Oh, this movie is

finally out. Maybe I'll go download it and watch it without thinking about all the parts that involved in making it show up." streaming sites really did a number on groups with the mass availability of everything. Somebody's in mayor saying that I need to switch example. All right, neutral corpor corporate frog. I like yes. Do you ever think where fruit comes from before or you do you just eat it? Like one of one of the core challenges that we face in open source is that is it's sometimes taken for granted as it as if it sprouted from the ground fully formed and wasn't the result of a painstaking selfless task of building and more than that wanting to share software. People

don't know where their open source comes from or even that they're using it sometimes. And let me just be very clear that's all right. But it forces some of us to hold up the front on those matters and make sure that the people are not forgotten as part of this process because open source is made of people figuratively not literally figuratively. Open source is made of people's hopes, dreams and selfless obligation to contribute something for the greater good. So let's talk about that. How does open source come to be? And for that we got to think about well again people. Not all people of course like this is a graph of the world's population as you can see it

continues to grow. Some of those people are probably joining GitHub right now at this very second that it passed. While the world population keeps increasing and the amount of packages that we have keeps increasing as well. Maintainers are keeping pace with that and maintainers are what makes things go round. More explicitly, maintainers are like Daka and you always need more Dhaka. Demand for maintainers of packages outstrips supply by a lot and that creates one of the core challenges that one of one of many core challenges that we have nowadays in open source to provide a more direct example here. So look at these two two people. You likely recognize one of them right off the bat.

Curl is one of the most if not the most popular open source project. It's well funded, well operated and its leader maintainer has been widely acknowledged for their contributions. Daniel or as he is known badger has received a lot of accolades for the work that he's been doing in terms of open source. On the other hand, we have our sync which probably as we speak right now is like syncing a bunch of content and packages from this from one side to the other and it's essential to how many servers back up their contents and yet has a single maintainer Andrew. How many of you recognize him? No hands. Good. Because indeed that's not Andrew. It's just some AI generated

face. doesn't change the fact that maintainers end up being underappreciated and underrecognized unless their projects are out there and very visible in terms of how they publicize themselves. Glad I didn't have to use this def. Okay, so maintaining a package is quite challenging in itself. Lack of maintainers is not a new issue. It's something that is as old as open source itself. Right. In the beginning and still as of now, it requires a lot of selfless obnigation to like volunteer your time. The in terms of people being professional maintainers, there's not that many of them out there. There's still a lot a vast vast swath of open source that is maintained by people simply volunteering their time to build

packages. As you can see there, that's a thing that ends up being mentioned. Why don't you have something shared on GitHub? Well, I have nothing on GitHub because I mostly write code that makes money. Open source tends not to make money and that is indeed a situation that contributes to this crisis in terms of having maintainers join us. So, economic pressure is one of them. Lack of incentive is another in a sense. How many of you here would be compelled after this talk to go uh go look for a package that is without a maintainer? There's a number of sites that aggregate these and pick it up. We'll talk a bit more about that later uh down the

presentation, but you probably are thinking well this weekend I this weekend I have things to do and then there's that thing at work and then the quarter is almost over. It is hard. It does require a lot of dedication. There's also lack of support structure. For example, understanding how can you become a maintainer? How can you just walk into a project and start making your contributions? There's some guides to that, but it still requires that you put a concentrated effort over it. And in the end, sometimes, as I said, regular life gets in the way. Like I work in open source, but once the clock hits six and I've done my 40 hours a week, do you

think I spend my weekends maintaining packages? Well, I do, but that's a thing that I do. I don't expect everybody to just go, "Yeah, I'm going to spend my extra time doing more work in a sense. I spend your weekend doing other things." So indeed maintaining an open source package requires a mix of dedication, strong knowledge, availability and selflessness. And we have some metrics uh that also tells tell us that like you can go to something like Rupology the packaging hub and you can see that for example that's a sort of the top 10 list of projects that they track by list of maintainers. You see at the 10th there's like 500. it start it starts dropping

significantly after that. Most of the things that are listed have one two maintainers. If they have three, we're happy. Like Lua, which has been used to build a lot of very important software. Hades being one of those, I believe, fantastic game only has three authors that do all of the maintenance for it. It's amazing that they can make it work but for large scale projects it is something that is perhaps difficult especially if you're covering a lot of things here is clearly a passion project and I wish them all the health that they continue it the other thing is like for example um like Rupology is a good example of this but Rupology is not uh a

complete repository of all of this information it's very hard to track the entirety of open source as I mentioned about the packaging environments the moment moment that you think that you have a complete record is the moment that something new showed up and you're no longer accurate. We currently don't have a really good method to have like a complete picture of open source. We have some data here, some data there, and we can make educated guesses on what the status is. It is clear the case of like the lack of maintainers. But for example, if you were to ask me how many people currently right now maintain at least one package actively the actively part is going to trip you up because

it's very hard to determine. Okay, perhaps they are associated with a package that is crystallized and doesn't need any new maintenance. Perhaps they're juggling 50 different commitments and they haven't had time to get to this one yet because it's non-prioritary. determining these things and how we measure and how we measure them to get like an accurate um result are challenges that we still face. The district face it, the ecosystem faces it like even all the proprietary ecosystems face it. Um and yeah, it it puts us in a situation where we know there's something missing, but it's very hard to quantify exactly how much. So, let's talk a bit more about this thing about maintenance. Sometimes there's a pernicious incentive

to contribute anything at all. Like saying, "Oh, you should do some open source contributions has been parited as a way to find employment." But it it it ends up creating an idea that oh you just have to put something out there whatever it is which creates challenges for the projects themselves that are trying to let's say on board 100 something new developers that are coming in every day because their project got flagged saying oh this is a great new project to start start throwing out commits but later down the line I'll talk about some of the new new challenges that are uh showing that are being presented by that other times even if you do contributions

you might not even secure a job using the very thing that you created that's Sebastian Ramirez he created fast API and you of course couldn't get a job because it required four years of experience with the thing that he created it is um it is of course very tongue and cheek there but it it does reflect a situation that one of the core things with uh being an open source maintainer is that perhaps unless you're looking for a job in a company who's well in that business, it may ve be very hard for you uh to get employment by using that open source maintenance experience as your foot in the door. I have this example here. I believe that

one of the other examples was the fellow that built homebrew that also had a similar experience where he interviewed a company and they went, "Oh no, that that's not going to work out like you built your own pet project." And that's a thing. For those of you who are not in the know, homebrew is very much how you do an AP package installation experience in Mac OS. So as a whole, maint maintaining an open source package is well, here's an artist description of what it generally looks like. This is barge haulers on the vulga from Ilia. It's very much pulling things along um at great cost. So now that we've gotten into into that mindset,

what do you think, and this is for the crowd, what do you think might might be the number zero maintenance burden for packages? Array start at zero, folks. Does anybody want to hazard the guess? This is a security conference. It's a big clue. Well, like fleas on the dog, CV numbers. Right now a lot of projects are getting like CV assignment has continued to r to rise and we've seen a massive growth last year in the number of assignments like you can see the the jump from 2023 to 2024 and we're on track to beat that number extensively in 2025. So more and more things more and more security issues and things that look like

security issues are getting identified and placed into projects and those become the number one burden because yes it's really nice to build new features. Yes, it's really nice to like get something new out there. But if you have like three or four CVs assigned against your project, you those jump the queue in terms of priority. like features have to take second place while you try to address this lest your passion project or your corporate project or something of the sort become the become something that is no longer favored because it's like oh no they aren't fixing the CVE so we got to move the things so being very clear having more CVE ids assign is a good thing

although there are some still some issues to iron out for those of you that don't know what DVWA is it's the damn vulnerable web app which is a training tool for um while teaching about vulnerabilities. Somebody somehow was able to put put up an advisory against it saying oh no this training tool that is purposely vulnerable has a vulnerability. Who would have guessed? It is indeed a surprise, but as I said, we still have some kinks to to to widen out, but we are working collectively to towards it for to to have CV assignments be verif verified, wellverified, and to uh crunch down on these records that just show up and like make no sense. Assignments as a

whole are growing fast like certain CNAs lead the pack. It's patch stack things like patch tack word fans or my dear favorite kernel.org who creates as working in a dro as you can imagine it's quite an interesting challenge when the kernel.org is just pumping out about an average of 14 CVS I believe these were Greg Slav's numbers a day and we got to address all of them across all the kernels that we carry. Indeed, CV assignment grows as package as um as package numbers grow. So the more packages you have stands to reason that you'll have more CV is assigned to them because it's linearly more you have more codes. And if and if you have more

code well you're going to have more vulnerabilities since the midst but while these are riding the same wave what about the whole ocean that sits behind it? like have we are we growing in capacity in terms of being able to resolve these because identifying a CV is one thing what about being able to address it so Jerry isn't here but this is from one of the things that he that he built he s couldn't attend this besides this is from cna pulse.org that he built and it shows exactly what we're saying like there are CVs that are growing in the number of things that they are assigning the like word friends and patch are believe going to be the

number one and number two and we keep getting more and more things but yeah the vulnerability reporting being on the rise isn't being met with maintenance and that is something that ends up on my desk as somebody that does vulnerability management for a show a lot of times we have to talk with maintainers we have to talk with upstream projects to try to figure out solutions for issues that affect the distro and are coming across uh our desk in terms of yeah we got to solve this because we're installing millions of servers across the world we need to get things sorted out so I'd like to share some small anecdotes from my experience as I said it's anidotes

it's not data we don't really have much hard data about this but they do reflect some experiences that I've had in terms of dealing with these things some are simple some are more complex and they are not hard guidelines. They're more of a pirates code of things that I'd like to see happening. So, let's start. So, one of the things that is really annoying like when you when you go up to a project and you you plop down an issue in plain view with a full reproducer when you know for sure it's a security issue. It's one thing to like put down the logs and say, "Oh, I found this thing. I don't know what it

is. Is a capture." The other is I already know that this is a security issue and I'm going to plop down a rep producer. If the project doesn't have a way to like hop on that very quickly, it's going to create an issue because like yeah, you people that are malicious actors are paying attention to these things perhaps more than project maintainers because they're very busy and they're going to start ways to weaponize this. So please, if you have one of these things in hand, see if the project offers private reporting. It's something that you can enable on GitHub or many other source management things where you can discreetly message uh the people in charge and say hey I have this

finding I think it might be security related can we discuss this before we go public with it if they don't have that enabled for some reason try alternative means of communications because before disclosing openly as a last resort in essence uh use all possible avenues of communication before just plopping things down there because it doesn't benefit if anyone except the attackers when well here is the full reproducer for your thing and they start immediately weaponizing the exploits. Next up fix this for the projects fix things as quickly as possible issues CV assignment somebody will figure out eventually we had some we're in a situation dealing with this with a project uh that was um that shall not be

named better in this case but they said oh this bug has been fixed some point between the 2.4 and the 2.44 44 releases. It was, I believe, a 9.5 CSS, a critical finding. And indeed, it had been fixed between those two releases. But if you diff them to see how much had the project changed between them, it had 1,667 commits, more than 5,000 files change, and no CV reference as the CV was published way, way, way after the thing had been corrected. This of course created the challenge of we know there's a vulnerability. We know that it has been corrected but now we got to go digging and try to find the patch because when we do patches and when we

do backports our goal is to like just pull the minimum amount of necessary code rather than pull all the all the changes in and this of course created a challenge. So the team was able to do it at some cost. So the idea here is that projects please use CV ids to communicate what has been fixed where and how. It might require you to go back and change a message or put a note somewhere that says oh this commit actually fixed this thing but please make everyone have a clear understanding of where the issue is, what's its impact and where it's emanating from. because that's how we get a good idea of exactly what is the

minimum set of fixes that we need to pull so that we can pull them into the versions of the packages that we have without being disruptive. Next up, let's say you're a reporter. Send in a fix immediately in public in a public PR. Be cryptic. Don't say what you're doing. Say literally this the yonom for this this isn't good like because this can create an issue. Let's say that the fix that this person is proposing goes against how the project wants to fix the issue. Let's think about the the talk that we had about the time the 64-bit time thing. Let's say that this person detects an issue and rather than saying, "Oh, I want to do I want to change

things to use the 64-bit strct." No, they just remove every single time call and go no code, no vulnerability. No, that's not how these things should be done. If you are submitting a PR to a package and it's like your first time doing it, work with the maintainers if you wish to contribute to fix. Agree on a strategy for it and how to test it. Make things things are properly identified. So you still align like unless you're going to take over the package and you're going to maintain it for the foreseeable future. It is uh a good thing that you should do is to align in how the the the core maintainers that are going to be

handling your fix after you leave are going to manage the package going forward. You don't want them to be taking on technical debt. Again, one for the projects. If you get a report from somebody, maybe an enterprising researcher here in this room, keep all information to yourself. Loose lips, sink ships. And this might seem sound on the surface, but let's consider how much of open source operates, right? There are situations where due to licensing hazards, projects that are well semi-open source or something of the sort or have particular license associated with it may be forth and they may have prominent forks. If you are running a project that gets a report that you think, oh, this might affect

this fork of my project and you can determine it because you know where the the issue with the issue was introduced in terms of commits. Well, you should at that point consider sharing this with your fork because it benefits the community as a whole. Hoarding information does not help anybody because you're doing this now and they might return the favor later. They should return the favor later. We should be sharing this information across forks so that in the case even the the mainline package that exists and the corresponding fork all march towards security. And finally, this one is something that I have to deal almost every week. You people making unreasonable demands. People need that CVE so it shows up on

their CV right now. The market is very adverse in terms of hiring. And I want to be I want to say like I found 50 Cvees. Please, please, please grant me a CVE right now. And if you don't do that, I'll publish this on my blog next week. It doesn't matter. that is a very complex project that involves a lot of people that it has a whole release cycle. I want this now. And as you can imagine, this is something that we end up getting a bit of in the dros and we have to diplomatically manage in terms of expectations and how things are going through because yeah, like for example, we release a new version of the DRO

every six months. Like if this if it's something that has to wait for six months for the next version to come out, it generally isn't, but let's imagine we have to be able to manage that and not disclose things before time. So in that case, we should Reporters should consistently set reasonable timelines based on the issue. It's a if it's a very complex issue that affects core parts of whatever the thing is that you are reporting it against. Let's say that it's not a oneline fix. Sometimes people do report things that require full architectures to be remade because oh yeah, we didn't think of this specific scenario. You should be able to set adequate timelines like 90 days

disclosure dates or even further work with the maintainers not against them and protect the community at large. People think, "Oh no, I'm going to adversely disclose this. I'm going to put this out in the public because the public needs to know." All right, but what did you achieve with that? Oh, now the public knows and they're safer again. How? Well, are they going to write their own patch for their own devices? Well, they could. It is open source, but that's unlikely to happen. What you just put out there is like a flag for malicious actors to see, oh, look at this new thing that I can race against the patches to start exploiting. Adversely disclosing is almost never a

good option. We should always be working with maintainers to try to get to a point. If you run out of avenues with the maintainers, like for example, the maintainers are saying, "Oh, we don't see this as a security issue." Escalate a bit. try to go to a district that like um distributes the package and try to see if you can get an agreement there for their version that they specifically distribute to carry a patch for it. Like you you're not out of avenues if the maintainers say, "Oh, I don't think that this is a thing." Right? So these are five things that I really like to see more of happen in my day-to-day. But do you hear this?

Oh, look. It's AI ground reminding us that we should talk a bit about AI given that this has been quite a trend and how AI is going to fix all of these maintenance problems for for us. Before explaining how AI is going to fix fix all these maintenance problems for us, let's talk about false positives. False positives come in various shapes and sizes. They can be issues just not security issues. There's a difference obviously there. they might not fit into the definition of the maintainer uh of what they consider to be a security issue based on their threat model. Uh like for example people uh say oh um I believe it's the the bin utils package.

Oh if if we put untrusted input in this it can be very dangerous. Well yeah the threat model for bin utils is well you're compiling code and you've already agreed to execute it. So yeah untrusted input is always going to be dangerous. Like if you report something like that it is not a thing. It's part and parcel with how the package is designed. And it can even be well nothing at all. And to this I say hi to all the curl hallucinations. We'll talk about them in a bit. It also might be difficult for an understaffed project to contest a CVE or muster the motivation to do so. Like sometimes, as I said on the farming side

of things, people assign CVS against projects without their consent or knowledge simply because oh yeah, I found this thing. I want to get a CVE for it. and that gets levied against the project and smaller projects might not have the capability of like redressing that situation. You get a CV there, it's there forever and these false positives inevitably create compliance issues down the line. This thing here, this is the response to what is a a well very well-known Nessus plugin. it's the open SSH PCI disputed vulnerability is that if you're running PCIDSS this is probably going to be flagging in every single one of your servers and it's always the same thing like open SSH does not believe this to

be a security issue but the CV got levied against it and they don't get to remove it talking about the the LLM hallucinations this is the list of uh bug bounty reports that are pure slop that the curl project has received in essence what that seems to be happening is that AI is pouring a bit of fuel and accelerating the rate at which these can be generated. It's got to go fast. You throw a piece of code into the LLM and ask it, hey, um, find security issues. And of course, don't forget in your prompt, you have to say, make no mistakes. At the point that was bug bounty, but now we're getting to a point

where the reports are a bit more structured and a bit more vetted. But that carries another level of consequences. These are just like some researchers creating an account and sending things um to hacker one. Well, okay, no big deal there. But here's something that just came in the news I believe this past week and it's something that we should be paying attention to. MFMPEG, which is a very very popular pro um project and that basically has a goal of uh being able to play every single um bit of video that you can find on the internet is facing an issue like Google has uh a bunch of programs running that are trying to find bugs in open source packages and indeed

they are finding a lot of bugs in ffmpeg. FFM Impact generally tops the charts in terms of the density of CVS getting assigned because well it's a lot of video codecs deals with a lot of input and output and as you can imagine that's fertile soil for vulnerabilities. In this case, um the vulnerability that the the straw that broke the camel's back was that um the AI agent found a very let's say peculiar bug uh that the coding Lucas Art Smush Smash codec espec especially the first 10 to 20 frames of Rebel Assault 2 a game from 1995. There's a bug there in FFMP pack. Do you have you did you play Rebel Assault 2? I

see some hands there. As long as the light Yeah. So yeah, it it is a thing. It's like, why are you sending me this? Indeed, that's the case. Why are you sending me this? Google is sending this to FFmpeg, which is a project made of entitled volunteers. And then there's an expectation of, oh, but we're reporting this. You're going to solve it. Remember, unreasonable expectations sometimes put pressure again on maintainers. They take security seriously. This is acknowledgedly a security bug. But the case is that in this uh the case that we are finding is that well Google is sending in the bugs. What about patches? Oh no no no no no. We don't we don't do that part. We just

find things. The LLM is good at that side of things but everything else well and this again puts pressure more pressure on maintainers. Here's another example. Uh this is from Nick Woffer. A he put out this message uh six months ago at the time that I was uh that that I took this screenshot. They maintain libxml 2 and lib xslt and basically they were finding the same thing like people were submitting issues on mass to their projects some detected with AI and basically putting some incredible expectations that took the priority from feature development and improvements is basically hey we're submitting these things please fix this these are critical vulnerabilities these are CVSs 10.0 Oh, these are being reported in the

open by filing issues against the project. As soon as they are filed in the open, everybody's aware of it and then it puts pressure on the maintainer to address that. In the long run, as they say there, putting such demands on open source maintainers without compensation is detrimental, especially for projects like these that have a single maintainer. But but more on that later. Again, the whole thing about oh, how AI is going to like help us do this, it's still an open question. We know LLMs can predict the next stat statistically like likely sentence which was work works well for code that has been written before like if you're writing a for loop it already knows what's coming next if

you're even if you're using like bad names or variables it's has to be I for the loops it can also identify issues by checking if something is not the most likely word after a sentence but the jury is still out if it can produce novel fixes like it can detect the thing and it can also suggest fix for it because if it's something novel that hasn't been seen before. What is it going to base its fix on? And how confident are you on those fixes? Consider also a world where AI runs all open source projects. Let's say that we solve this issue. We have a general purpose model that can do all of these things. What would that world be like?

Like if we had a commodity models that can fix things and just continuously improve on the code, implementing the features, finding and fixing the bugs, all of open source could be maintained forever. Well, doesn't that sound like a pipe dream? Yeah, it's something that we still don't know. And indeed, we don't know yet. It's something that is still emerging and we don't have good answers for these questions now. Can AI detect things? Yes, Google seems to be doing it adequately. Can AI fix all the things? We don't know yet. And whoever says they do probably has something to sell you. Speaking of that, uh, minor rant. Please stop pitching me supply chain silver bullets. Lots of things run on Ubuntu.

Unsurprisingly, Ubuntu also runs on Ubuntu. I am the supply chain. Like, don't tell me you're going to fix things. It's just me. All right, let's now that we have addressed all of these issues. Everything is awesome. We made it. Oh, hi Barbie. Let's not forget Barbie there. Everything is awesome. Like, we're going to go to a future where AI is going to solve all these issues and everything is going to be great. Uh yeah. uh do you guys think about dying? So we get to the part of the talk that I like to talk that I like to call what happens to the hole when the cheese is gone or more essenti more precisely what happens to vulnerabilities to

packages that simply well die. That is indeed one of the one of the other challenges that is much less spoken about because you talk about the things you see. Oh, this thing released new version. This thing released something something. But what about the packages that are still in use and like somebody forgot them? Forgot the face of their elders. Well, here be dragons. This is from a while ago in March. So, there's um a libf2 library that appears to be the standard way of parsing SPF records. This is email records in C. But its development is mostly stalled in the project prep. does an un unmatched pull request claiming to fix a use after free

bug and nothing happens from a use of free right nothing ever bad happened after a use to free well in this case it's a language it's a thing for parsing SPF records in C it's probably not in many places but it's still a thing that is there in the repo you can see the date there 2016 and crickets so far it's 2025 I think I checked this yesterday still there are in the same state Another example, 2025 again, of course, and something that was reported and supposedly fixed in 2016. Oh, look, it's still there, right? What happened here? What happened with this project? Like, this is this is the comment from 2016 saying, "Oh, I submitted the fix. I

tested it and no changes. The bug is still there." and somebody at the time this was in uh March um 20 April 2025 ran into it and ran git bisect and saw yeah all versions of this are still affected the bug is still there it's publicly disclosed and what's happening here if you want to take notes of the package names for you to check later if you're using this somewhere please I'll share the slides later so I'm sure it's fine like when you say this people just go ah I'm sure there's no problem these packages aren't in any dual repository Nah, that's the beauty of compiling from source. It doesn't have to be. You can

just search for it, go there, download the source and build it and anybody can add them to a repository somewhere. Like you can create your own pi package from this or you can create a pack a wrapper package around these. Nobody is using it. That's you don't know that we actually as I said when I pointed to we don't have good telemetry on this. We don't really know if somebody is using this. lib SPF2 is the first result when you search for C library to process SPF records. People know better than to use abandoned packages. No, people don't know better. Sometimes no alternatives exist due to circumstances such as language constraints like if you need to

do this in C, that's the package. So we should assign a CV for it. Then I don't we don't know like we don't know if we're not like creating a new false positive. It says that it has a use for free. Do we have enough knowledge about the package to go and check? Do we have enough knowledge to basically make a determination if it if it's correct? And we should anyway. Well, yeah, sure. But how do we we assign this? It gets assigned to that. And then what what's the path forward given that the the package is dead? And to say to talk more about these things, these packages are dead. This is a graph from Source Forge.

This is a study done in 2012 that shows how many how many projects in Source Forge were active and how many laid dormant in a difference of uh two years. And I hope that you see the irony here of me picking Source Forge because the whole thing is dormant right now. Like when was the last time that you used Source Forge? Raise of hands. Was it in the last year? No. Source Forge as a whole is full of dead projects that are facing this. So we're mere search queries away from finding orphan issues and orphan projects that just might be security related. Again, just think about the last time you used Source Forge. There are still some projects there that

resist that haven't moved into to the behemoth that is GitHub. There there are a lot of stuff there even on GitHub. As I said, it's a developers account per second. Oh, I forgot to count how many we got during this talk. But there are a lot of accounts that were created. They committed things and they they they just died off. Perhaps they created a package that got some use. Beyond the repositories that are emails to domains that lapsed, messages to forums and IRC channels that have gone unanswered, just a lot of things lost into dev null analoges. There's a lot of stuff that has been abandoned and is just there for the taking with possible security

issues. And with these unconfirmed possible ma may maybe vulnerabilities, orphan projects can be far more dangerous than what they appear should you bring them into your home. And it's something that people need to be aware of and sometimes I'm not like and you think oh you're saying this this is just for show like this just telling the story about different projects you sure you pick some examples like h how frequently does this happen like it doesn't it shouldn't happen that much like the things that are in use should be getting attention right well no this indeed happens every day remember that message from six months ago oh look from September like well offer quit maintaining lib XML

2 quit maintaining lib x sltd as well because yeah the pressures that were being put on them were on due these libraries are used a lot and now they're on maintain which one of you is here like willing to pick up the mantle and go maintain these massive hunks of code that deal with complex file formats not me um but yeah it is indeed a problem that we just ran into recently because of this pressure and what are the source what is like one of the possible sources of this pressure well the cyber resilency act for example while it doesn't apply directly to maintainers because they are not for profofit they are open source maintainers it still results in non-due

pressure like it places the primary responsibility on manufacturers who integrate these packages into products that sell in the EU market so the manufacturers have obligations performing due diligence managing managing vulnerabilities and for providing sbonds for their products which include the open source components that they use. And this part about the managing vulnerabilities where is where the the rubber meets the road like oh like the the maintainer themselves may not be on the hook for this. But if I bundle your package into something think about something very common like open SSH or something like that is used to parse XML a very common for file format like lib XML 2. you are going to be like

if I want to bundle this into my device, my IoT thing that is getting distributed, I have to make sure that the vulnerabilities in it are fixed and I will likely if I won't become a maintainer myself because reasons I will probably exert a bit of pressure on the maintainer and for that I have to put up the mandatory reference to this XKCD that always shows up in these things a project some random person in Nebraska has been thanklessly maintaining since 2023. three. If you apply pressure to that, the whole thing crumbles. Okay. Okay. So, this is fine. So, what do we do now? Okay. Let's try to come up with some solutions here. We're going to

solve this problem before I leave this room. Spoiler alert, we won't. But you'll see. So, first up, keep a watchful eye on your supply chain. If you spot that project, call it out. like if you see something that is no longer being maintained, call it out and say, "Hey, I think this project might be gone and we got to start thinking about alternatives." Or on the other hand, you might also spot something that you thought, "Oh, this is no longer maintained." And it is. In essence, it's complicated. like while there are some clear signs that the project is dead and buried. So like for example an archive repository, a public notice of demise or something like that um message

from the maintainer saying hey I will no longer be maintaining this. There is no good definition of what constitutes a semi-dead project like a zombie project that is still there. no commits for x amount of time. Well, that probably doesn't work because just the other day I realized that Vixirron had laid dormant for about 15 years. There's a commit in 2006 and there's a commit in 2021 from the same person. So, the project just looked dead. It wasn't. It's like the dead parent sketch in a sense. We don't really have a good sense as again we don't have the better the the good telemetry to determine is this thing alive? Can this thing still be

maintained? Should we be able to rely on this for the future? So watching your supply chain is good up to a certain point, but then we kind of have to figure out our own uristics on what what are we going to trust to put into our software. Next up, it's contribute sponsor a project or two. Like there are things like thanks thanks Dev or the GitHub secure open source fund or even the thing where you see on GitHub profiles where you go and go and put a sponsor thing or coffee or uh like just send some swag to people anything just provide some support of some kind preferably monetary because some of these people put these projects as head

oh sorry spoilers again it's complicated again um because in a wellorganized project with a foundation or a kind of figurehead is simpler to figure out where to channel the money. But take FFmpeg for example. Like how do you like it's a loosely group organized group of people that live in a bunch of different countries. How do you contribute money to like you want to contribute money to the thing, but how do you make sure that it goes where it needs to be and how do you make sure that it gets to the people that need it the most? It is very hard because there's no really know your customer process here for people that contribute to open source. Some of these

people are very mindful of their privacy and like you only know them by their handles or something of the sort and they might be the core maintainers or the people pulling most of the weight. It is not a simple problem where you can just throw money down something and expect it to come out where it needs to be to have the most effect. Lastly, well, it's the other kind of contribute. Uh what? Oops. My slides kind of seem to have worked a bit. But for the ah sorry I duplicated this. Um become a maintainer today. Like many of you when you leave this people are now locking the doors. When you leave this room there will be

people with sheets out there for you to sign up for maintaining a bunch of projects. They will be assigned to you as well as your materials to No, I'm kidding. That's not what going to happen. And as you'd expect, right, it is incredibly complicated to m become a maintainer today because of an incident that is is the XC utils incident. And for those of you not know exuttils was a situation where a state state nation actors still unidentified wormed their way into a very crucial consiliary project that would have allowed them should have should they succeeded to compromise every single SSH server pre-authentication and just have free access to it. So now if you are

approaching a project to become a maintainer I would hope that those things are held to higher scrutiny at least that is my hope because as far as I see it for open source security there's a moment before and after XZ utils because that that incident showed how close we were to having something as fundamental as SSH servers across all dros to be compromised irredeemably to a point where like how do you recover from this the every server that you're going to try to use to boot a clean image is already compromised by it. So yeah, becoming a maintainer is now being held to highest scrutiny which also creates a bit of a discouragement like you might

not want to disclose a bunch of information just for the privilege of being able to submit a PR to something and indeed execut more complicated. So, as I said, we're going to solve things, but no, it's technical difficulties. We still have these problems in front of us to solve, and we will be and we will continue to work on solving them. Some of them might be unsolvable, but we have to keep trying. Like, we have to keep trying to find a way to do a way of vetting maintainers in a way that is non-intrusive and allows us to trust that they're not infiltrated agents trying to do things. We have to find a way to get funds to

where they need to be to support people maintaining crucial projects that are doing it out of obligation because if this thing fails, the whole infrastructure of power plants or something goes down and they they they don't want that that weight on their consciousness. And it's also the case of yeah being able to identify things that have gone dead dead really dead and be able to move away from them in a clearly signaled signposted way saying this project is dead. You either step up to fork it but do not touch it further. Keep it as a me a momento of all the things that were done. So in that case spread the word. You are better prepared

for it now as you know a bit more about the open source space. And yeah, knowing is indeed half the battle. Thank you for t for your time and I'm now open to questions.

Hey, here >> I can't see you, but we're good. Yeah, there's a light shing over there. Yeah, I'll just um so we have in our nations um ways to is it >> I forget the words sorry uh when someone does does something wrong with software uh we punish them. >> Yes. Um do you believe that uh disclosure made uh with no concern for the maintainer for a package should be punished? Shouldn't we have regulations for that? >> Well, that ends up being a philosophy question on whether punishment should be doowled out. to be reminded of the consequences of your own action is not punishment. But it it does create a an odd situation like if you adverse disclosure is a a

complicated topic that we're not going to solve in this comment but I I do not feel that penalties should be levied beyond reputational penalties. Like if you are working with this person, they create a reputation for themselves that if if we don't acquies to their demands, they're going to just disclose out on us. So we can start doing the engagement going from there. Like we're still going to try to be diplomatic and try to do things for the best. Again, sometimes you can't think can fix things in 15 days, but in terms of legal penalties or civil penalties, I don't believe that should happen because it would create a chilling effect. It's just a case of

we're doing a lot of these things out of goodwill and like good understanding between each other and we and we should strive to keep a good understanding and goodwill between participants. >> Great. Thank you. We're just trying to be conscious of time. If anyone has further questions, please uh reach out to the Thank you very much, Dio. And let's move to the next session.