
[applause]
All right. Thank you everyone. Really glad to be here. This is actually the first uh besides in Chicago I've spoken at. Um find it a little bit amusing because I've given I think 30 talks over the past three years. Um but have not done one in my uh the town I live in. Um, yeah, this talk is all about open-source poisons, um, which I think have been a really, really long ignored risk. I'm not going to say I'm happy that it's being showcased so frequently now, um, because it's been exhausting, but um, yeah. All right. So, uh, open source contributors are heroes. They literally hold up the backbone of our infrastructure. The vast majority of code is open
source, right? 96% of digital infrastructure is from open source and technology is built on top of this. This XKCD is now 6 years old, believe it or not. Um, and it is truer than ever. um open source uh contributors hold together our infrastructure and I really recommend you look into contributing to it. Anyone can contribute to open source things. That's why it's called open source. You can go make a branch say, "Hey, cool. This is a feature I built because I thought it was something that was needed." Um most open source work is actually unpaid. Um, and while corporate sponsorship is fairly common, it is not enough to have people contribute to it as their careers. Curl, one of the most
used packages in the entire world, only pays one person, the guy that wrote Curl. He wrote Curl because he was facing an issue. And so he made a solution for it. And now hundreds of people volunteer for it, but he's the only full-time person contributing to the project. Um it was an absolute need and he went and created something to fill a gap. Um and recently they had to stop taking bug reports um and their bug bounty program uh their bug bounty program not bug reports because they kept getting way too much um AI slop and the founder of curl compared it to being basically DDoSed by all these bug reports. it was overwhelming. And he just recently had
access to Mythos, uh, the new module from Anthropic. And he said despite all the hype about it finding so many vulnerabilities, Mythos only found one bug in Curl. So he got thousands of dollars in access to the program because they heard about him stopping all these bug reports. and they found one after spending thousands and thousands of dollars in tokens. Recently, we've been under quite a constant supply of um supply chain attacks. Um it's been absolutely bananas. Um just this past week, TanStack um huge open- source uh series of packages um wormed their way into I think at least seven other major packages. um because of a worm that was introduced to uh their package and in
TanStack itself it was only live for three hours the issue but it managed to uh yeah my slides are doing something weird restart all right um
There you go. Okay. Um hopefully it stays like that. Uh yeah, so uh I wrote this talk long before all of these um before it was so frequent. I submitted this talk in November of last year, right after shy hallude one. Um there's now a Shai-Hulud 2 and I guess uh with tan stack it's going to be a shyude three. But um yeah, so I wrote this talk back then before it was just so frequent and I kept seeing all when I first started in security, I kept seeing all these open source packages. No one's managing it. No one's caring about what dependencies uh developers are pulling in and it was quite an absolute mess.
And most environments I've been in have been like that. People aren't curating or checking what developers are using because it's open source. It's fine, right? Um but clearly it's become quite an issue. Um with this package what happened was um there was a really really tiny uh change that didn't have any malicious code in itself. It just pulled in other dependencies using um uh the developer's own npm token and those dependencies were then malicious. So, it looked perfectly fine on paper and the developer that contributed to it, who we now know is a North Korean threat actor, spent months contributing good quality submissions, which made people a lot less suspicious about that specific person. Um, after it received that
package, it scan uh downloaded those other dependencies. It then scanned the device for crypto, for secrets, for anything it could, for new repos where it could self re uh populate itself into from that developer. And then it will also xfill all of that information in clear text um by uh just making a new repo with a specific name that they were scanning for um in a public repo. Um, since typically it's pretty easy if you have someone's access to someone's dev account, they can just stand up a new public repo. Um, Trivy, um, that was quite a long week. Um, Trivy is the number one reference vulnerability scanner, right? They're scanning for vulnerabilities and yet they were
compromised. Right? A GitHub uh, actions runner was misconfigured. It exposed a token and we'll talk about how it did that in a in a moment. Um, took that token and then used it to force push malware across a whole bunch of other tagged versions of software. Not just a new version, but reusing um existing tags for older versions, which helped really um increase uh the amount of uh runners that were pulling in that dependency. um that caused all these CI/CD pipelines all across the world to then pull in these malicious versions because they were only searching for that specific tag and they pulled it in and it spread like wildfire. Um Axios is hugely hugely popular um for
transmitting uh information over HTTP. It's the 16th most referenced package in npm. 80% of organizations are using it. Um and through a really really interesting um social engineering campaign, they managed to um make the core contributor of Axios think that they were actually helping him and that they were an organization that wanted to work with him. After working with him for months and months and being on fake Zoom calls with AI personalities, they managed to get him to install something that stole his GitHub pat. And then um they didn't actually end up um altering Axios at all. All they did was um add a tagged version of a new package um changing the tag of the existing Axio
version package to their new dependency. And then that new dependency would be pulled in. Um it was called plain crypto uh js. Um and it called that would then call a post install hook. Um which then you know searched for crypto and secrets. Uh one of the main reasons why these things are always searching for crypto is because this is a massive funding campaign for North Korean threat actors. Um and mostly organizations are not one of their largest targets. They're mostly looking for Vipackers who also have like tons and tons of crypto just locally stored on their device. But um it is a huge added bonus if you can compromise organizations. Um they then extracted their um older loot
and were really really sophisticated in how they were able to cover their tracks and erase any evidence that these devices were compromised. Um so what are the common methods these things um are happening through? Right? Stolen PATs. Please please just stop using PATS. Eliminate them as much as you can. Um dependency scanners um are another really really large um method of the attack. And then um infected GitHub runners. Um, there are so many issues with using runners on GitHub, especially with their default configuration. Uh, we'll talk a little bit about how really, really awful their default configuration is in a moment. Um, so fixed paths are a problem. Fixed longived secrets are awful. Stop using
them. Um, just get off of fixed secrets as much as you can. Everywhere you can get off of longived secrets. rotate them frequently. So many so many of these issues are happening because people have GitHub uh personal access tokens which have their GitHub permissions just as a fixed secret and they keep it for years upon years and then you know if [snorts] a thread actor gets that they get all the same access to GitHub that you have without any additional verification. Pats bypass SSO. If you have a pet, SSO is not being enforced. Um, dependency scanners. Um, so tags aren't immutable. They you can reassign a tag of an existing version to anything you want. A lot of lock files directly
reference um tags as what they want you to pull in. And so if you hit that tag and it happens to be reassigned to malware, you're then going to be consuming that malware instead of um having it directly actually linking to the package and verifying that it is the package that you want to pull. Um devs are often taught this as a method and it really means we need to cause a fundamental shift in how we are training new developers because they think if it's in the lock file, if it's tagged with the version, it's fine. And I've seen a lot of people say that it's perfectly fine if you know you have the version tagged and you're not pulling in
the latest version because so many of them are consumed by tags. You can end up pulling in these new malware even if you think you're using an existing safe version. Um, and Dependabot is a huge issue with causing all these compromises because it is automatically seeing there's a new dependency, there's a new thing and then pulling it in um automatically and then so many people merge dependabot automatically because right if your dependency is updated that's probably a security issue that's been resolved so you should update it, right? Not necessarily. Um, oh my god, GitHub runners. Um, so GitHub actions uses their own internal runners for running all of those scripted updates for any infrastructure changes
in your CI/CD pipeline. And runners, you know, when they're using environment variables, they're pulling those secrets into their memory. Um, GitHub runners by default have um passwordless pseudo, so you can just dump clear text Linux commands into a GitHub runner and dump the memory and dump all those secrets in memory right out of them. And this is constantly used in these supply chain attacks. You can easily disable passwordless pseudo by default in your GitHub runners. It probably should be the default honestly, but if you aren't doing that, you really really should. Um, and then you know they dump those commands, they scrape those secrets. It's not right. Um, so the development is development is totally
cool. There's no issues with it. We don't need to look into what devs are doing or question them or if if we talk about what dependencies they're using, we might slow them down, right? How how many people here have heard that before that they can't talk about that because you might slow them down. It's fine. They're using all these good dependencies in npm. Yeah, I it's interesting to finally have people start listening to me about that when I've been saying that in the IT security field for over 10 years now. Um I was like, "Wait, we can't manage that at all. We can't like have an internal library that we're using as a reference." No, no, that'll slow people
down. It'll slow people down if they don't have direct pseudo access to their developer machines. Um so I often see these things fail uh especially right now in two ways. Uh so category one um actively setting things on fire by developers doing things outside of their domain. Even not talking about citizen coders as some people might call it. Developers who are front-end devs might start building backend infrastructure. People who are on different teams might start trying to go branch out, add these features instead of coordinating with another team. That's not their specialty. So they're using Vibe code for it. And then they might be reviewing it and pushing it and following instructions from the vibe code engine. Oh yeah, it's fine.
Just stand up a you know a base 44 instance just for this quick test. And then you say why did you sign up and put our put development infrastructure on the service that we don't purchase and don't own. Um it's it's perfectly fine. Don't worry about. And then the category 2 is really high quality devs that know what they're doing are constantly being overloaded reviewing all of this vibe coded things from category 1 and they're they're panicking. They don't have a path to a good path for these people to push things into their workflow stream and build things in a cohesive manner. Um so what are the next steps I'm recommending? Um I really like Aikido
Safe Chain. It's free. It's open source. Um, it acts as a proxy for npm, pipole, a bunch of the others, and you can use that um to pull known safe actual dependencies instead of needing to worry about is this dependency been retagged or something like that. Um, please stop using PATS. GitHub apps um are the way forward. That is what GitHub strongly recommends over PATs. Um, SSH keys are somewhat better. You can control them a bit more, but not ideal. and then OAuth, OAuth, OIDC, all these other standards. Fantastic. Please use them instead of um anything I've mentioned before then. Um and you know, maybe don't run the latest version of everything. Um you can also
cryptographically hash and pin your dependencies in your lock file so that you're actually uh validating the Shaw of it instead of just using it by tag. Really huge step forward if you're using that. There's a lot of um there's a lot of really lightweight tools you can be using um in your CI/CD pipeline that will even automatically do that without you needing to go through additional steps to um specifically tag those versions using um the hash. Um revoke passwordless pseudo just ensure across your organization if you're using GitHub runners, if you're using any runners, passwordless pseudo is not enabled by default. It really shouldn't be. Um, and then scope permissions. Don't overprivilege permissions. You're probably going to still end up having
secrets somewhere. Ensure they are really strictly scoped for absolutely only what is needed. Um, because these all of these methods keep being abused to compromise and excfiltrate things. Um, continuous monitoring. Um, you can monitor your dependency inventory and what devs have locally. There are a whole bunch of tools to do that. Um, you really should be creating an internal SBOM of all of this. Um, really recommend it just so you can track it, keep on top of it and ensure that it's what you want to be ideal is happening throughout your organization. Um, and then you can use sea to flag and change um any components that are changing without um expectations um without you expecting them to.
Non-human identity is a huge issue. I really really recommend um you go and listen to Dwayne's talk later where he will talk about uh the difference between humans and sheep and hering and um how you really really should be managing um your non-human identities much more strictly and uh much more in depth than I have time for in this uh remaining section of the talk. um architectural visions essentially build out an SS uh secure software development life cycle you need an appsec team if you're developing software you need an appsec team security engineers are not appsec engineers it is very different and we often do not have time to go through all of these threat modeling and
everything else necessary to manage a secure software development pipeline and we really really need to stop relying on luck because there are so many so more uh sophistication than there used to be. All these sophisticated attacks are a lot easier to do with all of our exciting new helpers everywhere. Um so what does CICD pipeline stand for? Right? Continuous integration, continuous development. Security is not a point in time. It is something you need to continually continually work on. Security should be in every single part of your development process. um really recommend you implement SAS software um uh SAS DAS all those great things and ensure it's happening as close to development as possible right if it's in the PR instead of happening
before that that creates extra interruptions to developers workflow um which overall is not great always keep in mind your long-term picture and create a um healthier development process um if you have a developer experience team work closely with them. Um, I've been working on a production ready checklist with them and it's really been helping us create a vision throughout the company of how we want secure software to go into our pipeline into our product. Um, and developer experience is a really really great team to work with if you have them at your company. They work on things like standardizing how you build repos, standardizing your readmemes, standardizing all those really, really important things that make managing apps
so much less of a headache. Um, and also makes it much easier to onboard and have new devs work and know what they're supposed to be working on and how they're supposed to be working inside your company instead of keeping their bad habits from their previous um place of employment. Um so that is my talk. [applause]
Thank you. Um so I am a senior security engineer at SoundCloud. I am a parent of twins. I am also a founder of an organization called Cyber Polyglots. We're an organization for people that speak multiple languages in cyber security. We'll be having our first village at secretcon and hope to have our village appear at a lot more um conferences after that. Um I also have a sessionized QR code if you want to give me feedback on the talk. [applause]
You still have a few minutes. Do you want to take any questions? I'd love to take questions. Don't be afraid. We're all geeks here. Yes. Uh, so one of the things you talked about was not using GitHub pads. Um, I was looking at Go Releaser the other day and using uh like an OAuth flow with releasing a a release on GitHub and what it had you do instead was the GitHub app flow. But then what you end up doing is storing the private key for the app in a secret and that to me feels like it's just kind of recreating the same problem as a pat where it's a longived credential sitting in the secret is is
that in some way mitigating some other possibilities or is there a better way of going about that that you've seen? The question was let me get this straight if mitigating paths is a better mitigation for this sort of situation. Yes. Um, I'm not sure if that's universal or just for that specific uh Go release. I feel like some of them use other OAuth or other methods for GitHub apps. Um, but moving it to safer storage than on your developer machine is always a great step forward. Um, and then isolating um where you're storing it. Um, and then it also allows you to rotate it more frequently because then it's not directly on developer machines and you
don't have you have it in one place instead of every single developer machine. Um, but yeah, I I definitely think there are problems with that, but that's definitely a much better step forward than current state. Um, [snorts] I feel like I've seen other methods of it, not specifically for Go Release, but for other apps that I've seen. Um, but I'd have to look into that more. No problem. All right. And on that note, if there's no other questions, we've only got time for maybe one short one. All right. Thank you very much, Sean. That was a fantastic pro [applause] presentation.