Malicious Packages – they’re gonna get ya!

Hello, good evening everybody. Welcome back to B size Las Vegas second day uh at proving grounds. So this talk is titled malicious dependencies. They're going to get you and it's given by Mag Sage here. Uh a few announcements before we begin. [snorts] Uh we'd like to thank our sponsors especially our diamond sponsors Adobe and Iikido Security and our gold sponsors Formal and Profit. It's their support along with other sponsors, donors and volunteers that make this event possible. These talks are being streamed live and as a courtesy to our speakers and audience, we ask that you check to make sure your cell phones are set to silent. So, if you already haven't, this is exactly the

time to take up your phones and check if they're set to silent. If you have a question, you'll be using the audience mic that I'm holding right now to ask a question so YouTube can hear it. Uh, you just have to raise your hand and I'll bring the mic to you. As a reminder, the Bides Las Vegas photo policy prohibits taking pictures without the explicit permission of anyone in frame. So, I would ask you guys to refrain taking any pictures during the talk. These stocks are all being recorded and will be available on YouTube in the future. Uh and I will also uh advise please move forward if you're interested uh because you just get to hear better.

And with that, let's get started. Please welcome our speaker.

Greetings. Thank you all for being here and for being interested in learning about malicious dependencies. So, let's start off with a little bit about me. I'm a senior product security engineer at Pager Duty. Prior to that, I spent several years in software engineering and web development. I started out doing terrible, horrible things like PHP. Uh, my hobbies and interests include cosplay and house plants, and I love sharing knowledge and terrifying people with scary security stories. So, let's get going. All right. So first off, if you're only going to remember two things about this whole talk, those are them there. So that is that malicious dependencies have become a increasingly common threat and defending against them is complicated.

So those are the two points that I'm going to be trying to get across throughout this talk. Now what are malicious dependencies? So first let's differentiate those from vulnerable dependencies. So vulnerable dependency is one that has a unintentional flaw or weakness in it that attackers can then exploit to cause harm. So this is things like your cross-sight scripting vulnerabilities, resource exhaustion vulnerabilities, those sorts of things. And these do not typically cause harm all on their own. They require a third party to interact with them. This is unlike malicious dependencies. So malicious dependency is intentionally trying to do something harmful. In fact, it will often do something harmful from the very first time that you run it. It doesn't require

a attacker to necessarily interact with it. So, malicious dependencies are the ones that we are going to be focusing on here today. Now, where do we find malicious dependencies? Where do these things live? Well, they are typically found in the public uh package repository for basically any popular programming language. npm and pi pi are the ones we hear about the most, but at one point in time or another, they've basically been found in any of them. They can also be found in git repositories or operating system package reports repositories such as those for Mac or Linux. Now, what sorts of things do they do? What kind of evil evil things do malicious dependencies do? Well, they do

lots of different things. They might mine crypto. They might be ransomware. They may do a variety of things. So they may install a back door and then do system recon and then exfiltrate data. Or they may do very destructive things. They might start deleting directories or they may just completely disable a system. Another important thing about malicious dependencies is that they often target developers. They're not particularly interested in production systems necessarily. They're after your developers and the juicy data that they have on their systems. Things like API keys, credentials to cloud environments, databases, maybe some encryption keys. And these are also often stored in environment variables on developer machines. So here is a sample of a malicious

JavaScript dependency. What this does is it just grabs the environment variables and sends it off to a URL. So that's it. That's all you need to steal whatever is in the environment variables like your um API keys and whatnot that you probably don't want being stolen. Now what else do they do? Well, malicious dependencies often try to hide their behavior. They'll use various different kinds of techniques. So they may download secondary payloads that actually have the malicious code in them or they may use quite complex offiscation techniques like steganography using invisible unic code characters that was actually observed earlier this year. They may also do just boring things like B 64 encoding. Another thing that they do is often the

malicious dependency or library will also have a lot of useful code in it. So it makes it a lot harder to see that it's malicious when it's hidden amongst a whole bunch of mundane code that actually does useful things. They may also be hidden as subd dependencies of other packages. So you're not necessarily going to see them when you're installing a package and it is a subdependency of that one. Now how big is the problem of malicious dependencies? And the answer is big and it keeps getting bigger. So, Sonotype has been tracking malicious packages and publishing data on it. And this is the total number that they had found by the end of 2024. In 2023, they found 156,000

malicious packages in that year. 2024, 459,000 in that year alone. So, big increase. Um, it is important to note that Sonotype does sell a product that protects against malicious dependencies. So, certainly take their data with a grain of salt. Regardless, we can definitely conclude that it is becoming a big threat though. Now that we have a little bit of background on this, going to talk about the different kinds of tactics that attackers use to try to get malicious dependencies into your systems or your applications. The first group of these are tactics that generally apply when you are adding a new dependency. First is typo squatting. This is by far the most common. This is basically where

attackers publish malicious packages with names that are similar to legitimate ones and they're kind of just hoping that you will type the name wrong or you'll misremember the name and install their malicious package by mistake. This has become quite a problematic threat. uh last year just one example is a attack Pi Pi suffered where over the course of 10 hours 566 typos spotted malicious packages were published and they halted user signups while they were dealing with this and pulling them all down and on the slide there you can see the various different examples of the typo squatted names that were supposed to be for the pillow image processing library so it's quite uh quite an array of different typos votes.

Now the next one is Trojan packages. This is also called masquerading. This is where attackers publish packages that actually do something useful. They might be a parsing utility. They might be a utility for interacting with an API or crypto. And they're going to do everything that they say they're going to do. They're going to be a helpful, useful utility, but they also have hidden malicious features that you really don't want when you're, you know, using API keys or dealing with crypto. And the last one in this group is AI package hallucinations. So this we all know LLM make stuff up. They also make up package names. So basically in their code suggestions occasionally they will

suggest package names that don't exist and they'll actually tend to suggest the same non-existent package names multiple times and attackers will figure out what these package names that they are suggesting are and then publish a malicious package with that name. This has recently been coined slop squatting. So, and you can see in the example there, that's a screenshot from chatgpt and it's suggesting a ts migrate parser library that did not exist at the time of this screen talk. So, it was one that it made up and this issue was first documented in 2023 and it is still alive and well today even in newer LLM models. So, we don't seem to be able to fix it.

We reduced it, but we have not been able to fix this one yet, at least. Now, the next. So, these are tactics that apply to packages you're already using. These may be packages you've vetted, you trust, you've used for a long time, and you have no reason to think that a, you know, just a patch update is going to be malicious. First of these is dependency confusion which is a little confusing in itself. This is um where an attacker has figured out the name of an internal library that is used at a company and they figured out the version that's being used and what they do is they will then publish a malicious package that has the same name

probably completely different code but that doesn't matter. It has the same name and it's a typically a newer version and they publish that to the corresponding public repo. So let's say that we are awesome code and we have this application called my awesome app and this is the package JSON where we declare all our dependencies for my awesome app. We have this one dependency called awesome lip. It's an internal dependency. We use it here at awesomeco for talking to our other awesome apps. It's hosted on our internal resource and we're currently using version 1.0.0, but our package JSON is set up to accept version 1.0.x. Now, normally when we're installing or updating our dependencies, we're

expecting to grab awesome lib 1.0.0, we're expecting that to come from our private internal repository that's hosted on a corporate domain. However, an attacker has figured out awesome lib and they have published a package called awesomelib and it is version 1.0 one. So, it's a newer version and it's published on npm. And depending on how you've configured your dependency resolution when you're installing or updating packages, uh, your tooling may go and grab that malicious awesome lib from npm simply because it's a newer version. It doesn't care that the code is completely different. It's just interested in the name. The name of the package is the same and the version's newer, so it's going to grab that. However, this one is

fairly easy to defend against because when you are using um internal or private dependencies, you can specify that they only resolve from your private source and that they're never go that they should never be pulled from the public repositories, right? And the last one of these, this is package hijacking. So, this in my opinion is the most problematic to deal with. This is where attackers have compromised some account belonging to a package owner and then they publish new versions of the package now with malicious code in it. So this is typically going to be package repository accounts or GitHub accounts that are compromised. These are done in a variety of ways but there is a couple notable uh

methods that have happened. First uh this typically expired domains. This typically applies to older packages that say are abandoned, aren't being maintained, and the domain associated with that package has expired. So attackers figure this out. They register their domain and gain control of the account to publish new packages. And so suddenly you have a library that hasn't been updated in seven years and now has a new version except now with malicious code in it. Next one is compromised credentials. um standard kind of thing. Uh a lot of package repositories have started enforcing MFA on user login. However, this there's um this doesn't necessarily be effective for things like API keys that are not going to be

protected via MFA. So, those are still very useful if attackers can get a hold of those. And then that brings us to social engineering, which has some really fun examples to go through. So this one, this happened just a couple weeks ago. A very popular npm package, a llinter. It's important that it's a llinter. This will come up later. Uh it was compromised and now and briefly installed malware. So this was um the package maintainer basically just got fished. Plain old fishing email. Nothing nothing special. Just human error. And they weren't actually the one that realized this. It was someone happened to notice very shortly after the new malicious versions were published that something was up and they opened a

GitHub issue about it. And if you look at the actual changes to the package, it's pretty obvious that something's weird cuz now there's a new install.js added and a random DLL. So definitely very suspicious. And despite that, this one got caught quite quickly and new the token was revoked and new packages were published without the malicious code, there were still a ton of downloads for these packages. Even though, as you can see, there was only a few hours before the um the issue was resolved and newer packages were published without the code. So, super interesting, very recent example. And now another fun example. So xed utils. Exed utils was compromised last year. It is a Linux compression utility.

It's in a lot of very popular Linux distributions. And last year there is a backdoor installed into it. And this backd dooror would have allowed basically anyone with a specific private key the ability to SSH into the affected system. So really bad. We probably don't want that across a whole bunch of Linux distributions. And what's really interesting about this attack is that it was a social engineering attack that targeted the sole maintainer of the library. And um it took place over the course of three years. And so basically what happened with this one is as per usual there's a sole maintainer who's overworked and then along comes this individual Gatan who starts helping out with the library

and starts working on maintenance with it. And then these other accounts that were eventually concluded to likely be fake start bothering and pestering the original maintainer. They start bullying him and saying that oh he's not doing updates fast enough. He's not doing bug fixes fast enough. and that he should give uh Got 10 maintainer access. So he does and a little while later Gotan adds a back door into the code. It was hidden in binary test files. So it was very hard to detect and this got included in releases of XED utils and then those releases got added to the bleeding edge versions for various Linux distributions like Debian and Fedora. And so this is

kind of where we all got really lucky with this one cuz there was one engineer who happened to on happened to be on one of those bleeding edge releases and he happened to notice that his SSH sessions were taking half a second longer than normal. And so he delved into this and basically discovered and single-handedly stopped what probably would have been the worst supply chain attack in history. So, super super interesting and uh scary story. All right. So, now hopefully maybe I've scared some people a little bit. Uh let's talk about what we can do about this. So, what can we do to protect against malicious dependencies? And the answer is many things. There's a lot of things, but

there's no one perfect solution. Start off with some basics. So, education and awareness, you're all here. You know about this. This is also very important for developers to know about and be aware of because it targets them. There are some general precautions that can be taken. You can double check package names when you're installing packages. You can check package health. You can look at the history of a package or repository. You can scan the package code for anything malicious. However, these are not necessarily always going to be effective because of course there is human error. scanners are not necessarily going to be able to pick up well obfiscusated code and also the malicious dependency may be a

subdependency of what you're reviewing. So you're not going to look at it. And lastly looking at the uh repository or package health or history is not necessarily going to be reliable because attackers will often or some attackers will go to great lengths to make their their projects look more legitimate. So they might have fake community interactions, they will have fake downloads, may fake various things like forks, etc. to make it look more legitimate and make it look like it's not going to cause you any harm when it really really is. Now, another thing I have heard and read is that well, people think that they have they have a software composition analysis tool, an SCA tool, and they

think that that's going to protect them from malicious dependencies. And the answer is not not really. They're they're not particularly good for that. Um so SCA tools are commonly used to scan for vulnerable dependencies. So, they're typically looking for things with uh CVEEs, with GitHub security advisories. They're only going to be able to find things that they know about that are in their databases. So, only going to be able to detect known malicious dependencies. The other main issue with SCA tools is in how they are used. So in order for them to protect against a malicious dependency, you need to run them before you run that dependency. So coming back to the llinters, let's say I

am a um engineer and I have a linting tool like that one that was just compromised and I'm linting tools are basically just spell check for your code. They're very common to run locally during development. And so if I'm running that llinter and it's malicious, it's going to compromise my system. And typically developers are probably not going to be running security scans like SCA tools before they're running llinters, assuming they even run SCA tools at all on their local development environments. The other issue with uh SCA tools that's related to that is in CI/CD pipelines. So again, in order for them to protect a CACD pipeline, they need to be run first. So here is a sample and you can

see all these different steps where you have build steps, you have unit step, unit tests, you have integration tests and all of these things and these security scans which usually include sea scans are later. So or maybe they're in parallel, but they're not going to be in time to protect your CI/CD either. And one last thing about SEA tools is that depending on the tool, some of them don't scan dev dependencies by default. Dev dependencies are anything that is not included in the production release of your application. So that again is things like llinters, unit test frameworks, build tools. So, if your tool doesn't scan those by default, which admittedly a lot of SCA tools

don't because it'd be noisy, um it's also never going to be able to detect a malicious dependency that's in your dev dependencies even if you did run it and hide. So, I have said a lot of negative things about SCA tools, but they are still important and useful tools and you should still use them. They're just not particularly great for defending against malicious dependencies. Now, what else can we do? So, another option is EDR, endpoint detection and response, which is basically just fancy modern antivirus that runs on your computer. These are great for detecting known threats or say things that are interacting with known malicious domains. Uh they can do some they can detect some unknown threats based on

behavior as well. So that's going to work well for things that have really known unique behavior like ransomware, but it's not necessarily going to be very effective for things that are stealthy that are trying to hide. So this might be things that are say exfiltrating your environment variables and API keys and then blending it in with web traffic. Another one is private package repositories. So this is where you have your own internal package repository for things like dependencies that are say in the public ones. So like pi or npm. These allow for a lot of control. So you can control exactly what goes into them. You can make sure it's only approved dependencies. You can make sure it's

only approved versions and you can restrict your developers to only download dependencies from these and never from the public ones. However, maintaining all that is going to be a lot of work and it does really come down to how you configure these, how useful they're going to be because you can also configure them to just be a straight mirror of the public repository. So then it's just going to mirror all the malicious packages too. And the last one, so these are sort of a newer group of um of tools and they don't really haven't really found a common name for them, but basically what they do is they're a type of firewall and they will block malicious packages

from actually being downloaded. And how these work is that the companies that have these applications, they monitor the public repositories. They'll look for updates or new packages being um published and they will review them. they'll scan them for malicious behavior or look for anything that is malicious or suspicious and flag them. So if you're using their tool, then it will be able to block that malicious dependency from even being downloaded. I have never used these, so I don't know how good they are, how effective they are, but they are definitely a thing that is out there. Um, and this list was not exhaustive. Uh, I only had so much time. So those were just a few suggestions

that you can do. All right, so in closing, let's wrap this up and come back to a few key points. So malicious dependencies have become a increasingly common threat. They tend to target developers and these scrumptuous data that they have on their systems. There are many different tactics that that attackers use to try to get malicious dependencies into your systems and applications. And there are many different options for defending against them. However, there is no one perfect solution. So, as always, it's important to take a layered approach to security. So, with that, thank you for your time and I hope you learned something. [applause]

>> Great. I think I'm at time. I don't know. Do I have time? >> Uh, well, we're we have one minute. Maybe one question. And also, I would really ask you guys to please not use your cameras. Sorry about that. It's a policy. So, please do not. The link's on there. So, you could use the link. Please just don't use. Thank you. >> I do have a question about the domains. You mentioned about um domains that are obsolete or no longer used. I almost feel like most domains are owned by like GoDaddy, which is kind of old school, but you know, like certain websites, they own all the domains. So are they are potential hackers hacking those

domains and able to get those like information about which domains are available or not? And is it really those domains potentially or those websites that are kind of being the ones that are leading towards all the potential hacking and things like that? Um sorry I'm not sure I quite understand. Um for the expired domains that was usually like the domain that would have had the email address for the package account associated with it and basically the package is abandoned and the domain has expired. So so anyone new can register it. So just like any any domain that's expired. >> Um yeah it can also be a website domain. It yeah it would just be a domain that's

typically used for would have been used for email. Yeah, thank you.