Defending the Supply Chain: Building a Real-Time Package Firewall

Great. Okay. So, um, welcome. Uh, I'm going to be talking today about, uh, supply chain security and some, um, new ways to defend a supply chain, um, in real time. Um, I, uh, will be nerding out a little bit. Uh, I I offered to give this talk because mostly I just think it's kind of interesting. There's lots of weird quirks and things in the ecosystems that um make this a somewhat complex problem to solve. Um or at least finicky. Um uh I'm going to be walking through um kind of how we went about uh approaching the problem. I'm going to be talking a little bit about the the threats themselves. Um and then I'm going to be like basically dumping on uh

package managers and all the weird quirks uh that they that they have. Um so hopefully that's engaging. I'm going to be really diligent about um uh being on time on this. I haven't timed myself perfectly, so feel free to heckle if I'm like uh bumping up against the next talk. Um okay. Uh I am Dale. Uh I work for uh socket. Uh you'll see our booths here. Um may have maybe have heard of us uh otherwise. Um I'm on the engineering team um working on a a product that is a real time uh package firewall. Um I kind of have a background in uh dev tooling, building compilers, building frameworks, that sort of thing. That's kind of been

my jam. Uh more recently, um moving into the supply chain space and kind of the dev tooling around that. Um so that's my context coming in. I'm not necessarily a uh a security person. I am a dev that has moved into this and have some empathy for the engineers that um are engaging with these these systems that are hopefully keeping people safe. Um, all right. So, I'm going to start here. Um, every developer trusts their package manager. It's something they're interacting with dozens of times a day. It's part of their flow. It's not something that is like really even a conscious choice. Often, it's just part of what happens. Um, the package manager trusts what's in the registry. uh every

package you install um every dependency that's pulled in is uh is an act of trust. Um and uh because developers trust the registry to get their jobs done, so can attackers. Attackers can trust the registry to get their jobs done. Um so it's a really interesting vector. It's kind of like a hard problem to solve simply because we can't impose um we can't impose the constraints to keep everything perfectly safe because then it would be impossible to de develop software. So, um it's sort of like a a cat and mouse game where we really want to try to um uh engage with the the latest attempts to um to compromise these companies and these developers um while trying to leave a

path open for developers to actually get their job done. Um so that's that's my my angle um coming in. Um so the question is what happens when you run npm install? Um you're requesting code from a public registry. it's being executed oftentimes immediately either as part of the installation or or pulling down that package and sometimes at runtime. Either way, there is um kind of a threat that may be imminent. If you're running in dev mode and you have a software some software running and you pull down a dependency, it doesn't really matter if it's there as part of a post install step or if it's there as part of the um you know the runtime code

itself. Either way, it's going to be immediately invoked. Um and developers sort of trust this implicitly. This is like I said the bread and butter. Um so your package manager asks for packages. The registry sends them. Um your machine extracts these things and um just sort of implicitly trusts this arbitrary code that anybody could have thrown up on these registries. Um there is no checkpoint, there's no approval process. Um barring some security tooling. Um and the the code runs before anybody has read it. In fact, most of the time, as many of you know, nobody reads the code. It's just installed and pulled in. So, kind of like ripe for abuse, ripe for for compromise right there. Um, and

obviously this is something that attackers have uh been escalating as they've figured out how to how to weaponize this this trust. Um, yeah. So, uh I'm quickly going to go over the attack vector. So often times with these things there has to be some initial step where um the the developer is pulling something in either uh transitively or um themselves. So sometimes it's uh typos squatting um meaning that you're trying to type load dash or something like that and um you're getting a a different package because you pressed the wrong key or in the wrong order or something like that. So that's that's a pretty common one. Um account compromise. Now, this is where a

lot of these things uh are propagated through the ecosystems um is through account compromise. So, I'm going to be talking about that a little bit more um and dependency confusion. So, say you have a scoped package internally um and then somebody realizes through some leak of information that this package is used internally and it's not um it's not actually reserved in the public registry. Well, then you can go register that package and put something compromising there and people might pull it in if they don't have their computer set up perfectly, right? Um, so great example of this recently was Shy Halude. Um, 500 packages compromised through a chain of compromises. Um and uh we we have some some kind of uh

companies that are are of good repute that got compromised by this and that helped proliferate this through the the ecosystem. Um you have um maintainer accounts that were compromised through through this um kind of chain of events. Um, and uh, you have these patch version bumps that look benign, that look like any other thing that are automatically installed if you're just not using a lock file or if you're just happening to be uh, updating your dependencies. Um, either way, um, it looks benign and unless you're reading through every line of code, you're not going to be observing that this is happening. So, um, I thought this was cool. Uh, cool. I thought it was very intriguing how this

all came about. um really sophisticated the sophistication of these attacks has been ramping up um requiring some some different um ways of engaging with the problem. So, first you have like this um pre-install loader that when you this dependency gets pulled down, it uh it's starting to run arbitrary code in the background. So, it's not obvious. It's not blocking. Um there might be a process running, but unless you happen to be looking at every single process running on your machine if you're a you know masochist um you you wouldn't notice that, right? So, it's just running in the background and it's running in a privileged mode because these are developer tools. Um next up, you have the CI pivot. So that starts to

segue things into um extracting information in these CI environments that relate to publishing of packages. Um then you have the credential harvest. So that is um where uh I'm going to talk about this a little bit more but where the the secrets that are running on these machines are just kind of uh collected up. Um and then uh you have the propagation. So you have the um the uh tokens that are um uh captured internally um in in that like um harvesting phase. Um and then and then you're these secrets are being exfiltrated um in using some mechanism, right? Um and then there's ecosystem spillover which was a unique aspect of uh this particular campaign uh where you

know it's really fun to rag on npm and obviously that's where a lot of these things happen. Um but in this particular case it's spilled over into Maven. Um so it it's not a unique problem to npm per se. It's just that npm is wildly uh popular and that's uh makes it right for abuse. Um okay. So uh and and the word worm uh propagation kind of indicating that this goes back to the beginning now so that anybody that pulls down a compromised package can go through the same chain. And that's why Shahalude was so um problematic um is because uh through this process multiple maintainers in a chain were or in a tree I suppose uh

were um were compromised and then all these uh packages were compromised and then anybody basically installing with any project had the opportunity to be compromised. Um so I want to focus a little bit on this first step where the thing was downloaded something bad happened and that kicked off this chain of events. That's the That's the problem I'm thinking about and we're we're going to be talking about today. Um I'm already 9 minutes in, so I'm going to I'm going to skim a little bit, guys. Uh we're doing this live. Um all right, so we have the pre-install loader, and I'm not going to um go through all this. I'm just going to kind of put this up on

the screen. Um this was interesting because it involved bun. even if you weren't using bun locally, it it like pulled down a binary and things were happening behind the scenes that you may not even been uh familiar with. Um so so that was an interesting aspect of this. Um as probably many of you know once um when you're running these these things in CI there have to be secrets injected somehow and uh that's either for publishing for interacting with external systems or any of any of these things. And so once the uh the local dev machine is compromised or maybe CI is is compromised directly through a dependency update um there's this opportunity now to interact with these

secrets that are running in CI because the CI implicitly trusts the developer and the developer implicit trusts the registry and the registry is where the attack happens. All right. Um and then exfiltration. There were some really interesting things about this. Um, so I'm going to pause here and not not breeze by too quickly. Um, if you're not familiar, truffle hog is um is a tool that can kind of scan your local system and look for secrets that might have been exposed. Um, and so it's actually a security tool that in this particular case not only looked benign, it looked beneficial. It looked like something that should have been happening. Maybe maybe uh MDM pushes this out and and and runs it on local

systems so that um you know your IT admin can know uh if these secrets are being compromised. So there's an actual security tool being used for the compromise in this particular case which kind of it it brings a level of sophi sophistication that we may not have actually seen um in in previous campaigns of this kind anyway. Um so uh and and then it's just like it's grabbing all the secrets it can right um it's it's uploading it eventually um to a um a GitHub repo of the uh of the uh compromises uh account um and then um like it it uses some like very basic offiscation where it's like tripleB 64 encoding these things uh when it's

uploaded. So it's really not obvious. you might uh decode it and be like, "Oh, it's still just random letters and and numbers. I'm not really sure what's going on here, but doesn't look uh concerning." Um, so very very trivial, but um obviously good enough um level of obiscation with the actual exfiltration that went on. Um and then the more propagation. I'm going to I'm going to skim through this a little bit. Um things happened. It spread. Uh it was bad. That was sad. Uh that's a summary for for that slide. Um yeah. Okay. So uh and then the the cross ecosystem spillover. Like I said, this was kind of a unique aspect of this particular campaign. Um I I personally

was very surprised when I learned this. Like I wasn't necessarily on the front line learning what this was happening. I was just kind of reading the retrospectives and this was something that really surprised me. Um it wouldn't have been something that I necessarily expected to happen. Um, and notably if uh if your Maven packages are compromised, that introduces the opportunity for things like the npm packages to once again be compromised. So thinking that okay, you've solved the problem in one ecosystem means that you're safe is um is is an unsafe assumption in this particular case. Um all right, so point I'm trying to make here uh the thread is still evolving. Um, and in fact, uh, last week or maybe

two weeks ago, something in there, there was a new, um, malar campaign that was naent. It actually didn't really get off the ground, but it had all the hallmarks of Shy Hilude. Um, and it was caught early, which was great. Um, there were some really interesting things here that went on. Um, it it uh, they had were kind of setting the groundwork for um, for kicking off things like typo squatting and things like that as a way to start this chain of events. Um and uh it it uh it started interacting with things like with uh MP uh MCP servers, excuse me. Uh so like AI coding tools are now interacting with this thing. There was um I don't think I have it in

the slide here, but there was like Xfiltration through DNS which is seen been seen before but you know kind of novel and not necessarily even something that you're looking to protect against at the you know the network level. So just making a benign DNS request can actually be something that uh that helps to excfiltrate these secrets. So very interesting. Um here's the point I'm trying to make which cues up the second half here. Um postinstall scanners are not always enough. Now I'm not going to say that uh like real time scanners are enough either. I think we need to have defense and depth and these are complimentary tools. But there's a gap here that historically has

not really been addressed. Um, and so there's an opportunity to to protect against these things a little bit better. Um, we need interception before install. I'll leave it there. Um, okay. So this intro uh this is where I'm going to talk a bit about uh the uh real-time package firewall. So um let me just yeah I'll start here. um as I started with uh developers are um interacting with their package managers regularly on a daily basis many times a day um and so it's it's a frequent touch point. So what we did is we built this very basic wrapper around a package manager where you could prefix any given package manager command and it would do

things behind the scenes and I'll I'll get to the do things in a minute. um it uh will um intercept uh the the network traffic. It will check with our source of truth, the the mother ship um and determine whether this thing is actually safe to install. And it will do that for every single package request that is sent out from your package manager to the registry. Um and it works across ecosystems. I'm not going to I'm not going to belabor this. Um, so this is probably where most of of you all are interested. So I'm hoping I'm getting there fast enough. Um, starts with uh uh interception TLS interception. Um, we have the package

manager. It makes a request to the registry. Right in between is this proxy firewall where it's it's an HTT HTTPS proxy meaning it it handles the connect verb of of the HTTP protocol. Um so it has to be configured your package manager has to be configured to use this as an intermediary when uh interacting with the registry. But then all this traffic is flowing through it and I'll dive into that specific piece in um in a couple slides. Um so the the um the request is intercepted um you get just like this basic path you can associate it with a particular registry so you know kind of what what you're dealing with here and that can be turned into a

uh a pearl a package URL which is kind of a standard way of representing um a particular dependency in a in a unique way and um here we're fudging a little bit because there is a distinction between um artifacts and path packages. So like in the in the Pippi Python world, you might have multiple downloadable assets for a package at version. Um but uh by and large if if you know the the wheel is compromised, you're not going to want to download the other things either. So um we just sort of normalize it to the package. Um and then uh with that pearl, we go to our policy engine, the mothership as I said, and that determines whether a package is

uh is blocked or allowed. Now, um I'll point out the obvious, uh because apparently that's what I'm doing. Um there has to be a source of truth for this to have any benefit at all. So, um I'm not going to say like this just magically solves all the problems. There has to be some mechanism that is in real time or near real time identifying these threats and marking them as threats so that they can be blocked in real time. But the um kind of important aspect of this is this doesn't require any human involvement beyond maybe what happens in identifying that threat. Um there is a real-time aspect to this that as soon as a threat is understood, it is blocked.

Um and and that's that's basically the premise of this approach. Um so for uh to to give a little bit more insight into what's happening with the um with with the actual proxy itself. So, it's a little bit arcane um and it's actually been hard to explain. So, hopefully like the the diagram actually helps to explain here. There's actually four streams of data. Um there is the inner HTTP and this is probably what you're expecting, what you're thinking about when when you're talking about um a package manager interacting with a registry. This is where the uh package manager is is making get requests for tarballs and things, right? And then that's wrapped in TLS. Um hopefully uh

then in this particular case the proxyy's in the middle. So it is actually man-in-the-middle the TLS connection. It is doing TLS termination um at that point in time. Meaning that the package manager has to trust this proxy um in order for it to actually be able to effectively decode that inner TLS inner HTTP traffic, right? Um, and then that is all wrapped in this connect transaction, which is another stream of data. And then that is wrapped in this outer TLS because when you're talking between the package manager and the proxy, you're going to want that to be encrypted as well. Um, and so there's there's these layers of streams of data, um, these layers of abstractions that

actually make this a little bit, uh, finicky when it comes to implementing the thing in the way that a package manager can uh, interact with successfully. So, okay. Oh gosh. Okay. All right. Here we go. We're going to just breeze through the here. I'm going to skip over this. I did not time this well. Sorry, folks. Um, there's support for a number of different um kind of facets of this. There's different types of registries. There's private registries. There's all these different things. So, I'm I'm breezing past the stuff that is interesting, but not necessarily core to um to this. And we're going to skip Oh my goodness, this is great. C strategy. Okay. Real world lessons. We

made it. Um 5 minutes left. Yeah, we're cruising. All right. Okay, here we go. Um okay, so it works with npm is just the beginning. Like that was actually where I started um just making it work with npm. Um but there are a lot of ecosystems and we saw before there's crossover and we also want to um kind of impose these guard rails without they them becoming um super opaque and and actual obstacles to doing development work. We want them to kind of work magically, transparently without interrupting the flow, without being a problem. Right? So, um that's just the starting uh of where where we went with this. Um and I'm going to talk a little

bit now in the last few minutes about the uh quirks with some of these package managers, right? Um Cargo, if you're in um invested in the Rust ecosystem, is a fantastic modern package manager, but it has a quirk. Um if you specify that um a particular um uh rootert should be used when interacting with remote systems meaning uh if we man-in-the-middle the traffic this the the cargo package manager needs to know how to trust that traffic. Um cargo has some some interesting quirks. So, as soon as that happens, we're we're man the middleling some things, but we probably don't want to just get in the way of every traffic. We want to get in the way of of registry traffic and

inspect that traffic. Unfortunately, Cargo doesn't really let you do this. Um, it it just kind of like if you say trust thisert, it's like everything has to be signed by theert. There is no traffic that will be permitted unless thatert is in the search chain, right? Um so uh that's a problem because cargo doesn't just interact with the crates endpoint. It also interacts with uh GitHub. It interacts with private registries. It interacts with other endpoints that are maybe telemetry endpoints or different things like that. And then all of a sudden this traffic's not working. Um and that's a problem for cargo. So this was this was a wrinkle. Um yeah yarn. Yarn is anybody in the room have they worked on

yarn? Okay. I'm sorry. I'm I'm going to rag on yarn. I'm really really sorry. Um yarn is is a special snowflake. It has some really weird quirks, especially when it comes to uh interacting with a connect proxy. So, um it there's some benign things like, oh, it just deals with URL uh strings a little bit different. That's fine. Um the this is the weird one. This is the one that really got me. Um Yarn is trying to interact with registry npmjs.org. um we're saying, "Hey, here's the rootert that you can you can use to interact with that." Um that's like stamped with a domain name with a host name, right? Um but the outer TLS

connection, the one I showed you before between the uh the package manager and the proxy itself, Yarn expects that um to include the um the registry host name as an alt name. It doesn't make any sense. Um, and probably nobody's run into this because nobody's tried to exercise yarn in this way before. This is kind of a new approach. Um, and so it it makes sense, but uh we wanted to work without having to like change all the the package managers in the world. Um, because that was, you know, a pretty heavy lift. We're not going to boil the ocean. We just wanted to get it to work. So, this was kind of an interesting uh

problem to solve. Um, and there's some variations across uh urine versus the newer ones are a lot more solid. The older ones are still in heavy use. And it gets a little bit funny. Um, yeah, Gem, if if you love Ruby, uh, okay. Um, sometimes like just depending on like a patch version, y'all. Um, it will just decide, "No, I'm not going to interact with HB proxy at all." So, there's now like constraints that we have to communicate to people if they're trying to use these firewalls. um say you know you actually can't use four you have to use five of this minor version of you know so on and so forth because gem is just kind of like a little bit

weird and it also varies across um operating systems um because of how the like the SSL the TLS layer that it's interacting with and using for um fetching packages um fortunately gemspec requests which are um not that tarballs themselves but um they do respect the the proxy so it's actually internally inconsistent um in a way that was fortuitous for us where we could block that instead and it ended up doing the job. Um, yeah, and then I think I'm just about at time. So, I'm going to just show all of these on the slide. Um, we kind of have two ways we've deployed this thing. one is as a service deployed in the cloud and you can kind of just make sure

everything talks to registries through the service and then as I showed earlier we have a locally running um uh command that you can prefix your your package manager command with it'll spin up this proxy ephemerally it'll actually mint the rooterts and place them in temp directory for the duration of that process and then throw it away. Um and so to make this work when you're um kind of across both of those those paradigms um we have generally approached this uh especially for the locally running one the one that wraps it um we have approached that just through environment variable configuration and it it's a little bit of a zoo as I you know titled the slide um it does generally work

there are some weird quirks and things like that but um because this uh you know environment variables will propagate through the um process tree um it's even if you have npm commands running within npm commands and so on um everything still receives that ephemeral uh conf uh configuration for the process the firewall that's running locally. Um yeah, that was the last one on the slide. Um I'm at I'm at time so I'm going to I don't think I really had much else. Oh gosh, let me just revise that. Um, wow. Yeah, I didn't time this, folks. I'm so sorry. Um, yay. Okay. All right. Yeah. Um, yeah, I had wrap-up slides and we could skip those. Um, I don't think I

have time for questions. I want to leave uh time for the u the next presenter. So, um, if you have questions, I'll be out and and come and find me. Happy to chat.