Snow Nor Rain Nor Dependency Confusion: How to Deliver the Right Package

all right everybody Welcome to bsides about time to get our talk on the uh Road it is my pleasure today to introduce Jessica Smith and Justin angler we're going to be talking about uh dependency confusion take it away awesome thanks so I'm going to stand about here hopefully you all can hear me all right so uh as our friends the organizers mentioned uh we're going to be talking today about the complexities of the US Postal Service today that's why everyone is here right um no we're talking about uh the dependency confusion issue we're going to be covering what it is and op we did exploiting it and then at the end talking about how we fixed it so uh

first a little bit about who we are um I Jessica this is Justus he's off in the shadows over there we are Engineers on the offensive security team or red team at block uh which uh that team has existed for about 2 years now um and this next line is lawyer approved we are a global technology company with a focus on Financial Services made up of all these other components which you can read um so basically we're going to be talking today about one of the very first red team Ops that we ran when we were a young new team uh exploiting dependency confusion uh so first let's talk a little bit about what dependency

confusion is in case anybody is not familiar and just so you know we did not invent this vulnerability uh we owe a lot to Alex berson's uh great medium post which we're going to link at the end which uh sort of kicked off the vulnerability back in I think like 2017 so basically organizations can configure a package repository like artifactory to store packages and serve them to their developers and just a reference uh I'm going to be talking about artifactory uh throughout this talk but uh the same general concepts equally apply to Azure artifacts or Nexus zype or whatever package repository your particular company uses so basically when a developer runs a commands like npm

install or pip install on their work machine the package they get does not actually come directly from npm or pip because their machine has been configured to First query artifactory so artifactory will query npm or pip or wherever that package is supposed to belong and then it sort of serves as a middleman between the public package Repository and the developer so artifactory will then serve the requested package back to the developer one of the benefits of this is that you can also configure artifactory to look at a private package store so this allows you to publish packages internally and then share them within your company within your own developers without having to open source those

packages because who wants to contribute to open source that's gross um so uh the problem here is that the package manager can get confused o a name drop if there is a package with the same name in both the public and private uh package store you know the question is which package to serve in that case and if there is such a conflict generally the package manager will simply default to serving whichever package is higher versioned this leads to an attack where if an attacker knows the name of a private package they can publish a malicious package with the same name to the public Package Store like uh pip or npm or wherever and then when the

developer runs a command like npm install they expect to be getting the private package which they're familiar with but instead they get the malicious public package which contains the attackers code and this leads to remote code execution on whichever machine ran the install so while in theory this attack could work with any language that has a public Package Store like you know python pip Java maen ruby gems whatever uh the most common Target is Javascript and npm and this is for a couple of reasons first of all it's easier to discover lists of JavaScript uh dependencies uh for an open uh for a uh a front end language because developers can accidentally include the package.json file in the source map and

so then uh it's in the front ends and attacker can simply pull it out of the browser browser when they visit the website so they could even write some sort of web scraper uh to do that automatically for them whereas for a server side language it would be a lot harder because uh in order to get a list of package names it would either have to be open source or leaked somehow which is a lot harder for backend code and the second reason is that it's a lot easier to go from package install to rce in npm because npm packages allow you to Define this nice uh install time script so you can see an

example package.json file here which is like the npm package config file and you see this scripts attribute and this pre-install hook which is literally just a piece of uh script that runs when the package gets installed so super trivial to go from package install to rce as opposed to having to wait until the package gets referenced in the code or something like that and just a note it is common for Defenders or scanners or things like that to flag scripts as potentially malicious if there is such a uh custom hook in say pre-install or post-install so that's something to keep in mind if you are a Defender this is an opportunity for you know a warning to

raise or something like that and if you're attacker something to keep in mind uh if you do want to be extra stealthy baby don't take this route so now that we've covered sort of what the vulnerability is is in a generic sense I will pass it over to Justin to talk about the op we did exploiting it all right so the first thing that we had to do is figure out which package exactly we wanted to hit so um we went through we decided to do npm first because it was kind of the most realistic like Jessica talked about with being able to see sometimes publicly which packages are being used um so we used a script called confused to look

at all of our um internal repos to see which packages we had that would be vulnerable to this there were 984 so how do we narrow that down um we want to find something that gets used frequently and won't break in prod and we found this really convenient one that does linting of ownership of like which teams within the company you're supposed to own a particular file and checks to see if those are there so this is great because even if the lint check doesn't work like everything still works nobody's laptop breaks everybody can still do their work and so on so this was a perfect choice so um we did give this a try and

upload a malicious package to npm and they spotted it pretty quickly about a week um but this one was a very naive package it didn't have any obfuscation it did a very simple like call back and it didn't do any anything um so not surprising that it got caught uh so what are we going to do differently next so the first thing we want to do is make sure that we're putting this in a public place we don't want to be exploiting random people's machines we only want this to run on block machines so um we did some digging through laptops to figure out if there are any files that everyone in the company gets because you

know we have some centralized config things and there was a file um and it's the same on every laptop so this gives us a a great source to make sure that nobody else can run our code we look for the existence of the file hash it and then we use the result of that hash value as a decryption key for the payload uh borrowed from stuck net a little bit if there are any stuck net authors in the room thank you um so the idea here is that no one else should have this exact config in their file so no one will ever be able to run this um it's great because it prevents um prevents us from running

other places but it also um makes it more difficult to analyze what the payload does because it's all encrypted um this is the sample code here is just our our script that generated the encrypted payload yeah there's an md5 in there yeah I'm okay with it you can talk to me about it later if you don't like md5 um so now that we've got this package uh we have the payload working we have we're sure that nobody else is going to be able to run it but us um we're going to put it on npm and to make it be a little less obvious uh um we put in an actual description this time that

is um you know made up garbage about what this thing does if you look at the code like the the part that does decryption is talking about like a license file check or something um Jessica looked up uh some commonly used JavaScript code um from stack Overflow and put that in the actual body of the package so it didn't just look like an empty package um and this one never got caught uh we took it down when the op was over um but it seemed to work fine and and nobody spot it so this is a screenshot of our artifactory instance internally and you can see that on version 030 it's showing that it's coming from the npm internal

cache and starting on version 031 it's coming from the public source so this is where artifactory has picked up our malicious package and you can see that starting on 032 we actually got some downloads so um it took us a couple weeks we were a little nervous that nothing was happening for a while uh but it turned out that someone I think more than one team started a new project around that time and so there were a bunch of new people running the install scripts for this new project which included the lint uh owner out lint package which means that we were getting our shells we got about 20 shells um as Jessica mentioned this was

uh baby's first red team op for us and uh one of the mistakes that I made was that uh we put the payload in SLT and so we lost a few people because they rebooted their machines and then they couldn't execute anymore um we got our persistence by editing uh the Chron tab file and when you do that in the command line uh they'll be a little pop-up that shows up because the the user gets asked hey do you want the terminal to be able to edit the Chron tab file and of course because people are just installing software they accepted that prompt and just let it go um we should have asked for rud instead

and next time we will so we've got a payload running on laptops we need to be able to control it um we didn't want to install a bunch of things uh we wanted to have strong assurance that the um that people talking to our C2 server were our actual uh participants and not uh random people on the internet so we uh had mtls we wrote everything in Python both client and server side python did some mtls to make sure that it was a author ized uh connection and then we use that to do our CNC so um shout out to truffle hog I know you all have a table here uh we use truffle hog and a few other things to

try to find interesting things on these developer laptops to pull down we pulled down private SSH Keys we pulled that in the keychain files um block has this um housebuilt thing that allows you to see which people have access to which apps and which apps have access to apps and so on we downloaded that whole thing for later use on other apps um grabbed a whole bunch of source code um and we also got a a GitHub Pat so we could um masate as that developer later however um we were running out of time on the op we wanted to we had wanted to originally add code to GitHub that would cause there to be

malicious essentially our stuff running in prod um but we didn't have time to build that so instead we just tried to see if we could get something committed to GitHub so um Jessica was on somebody's laptop she noticed that they were editing a bunch of files in a particular folder that looked like this new project that they were building so after they had were done editing a file she jumped in and edited the file again and then they committed that and pushed it up to GitHub um and all she she put in there because we didn't have time to build something good was hey if you see this talk to offsec and you can see that the

the reviewers did see it so that's good um and they did come talk to us um so this was the end of the op at the end we were able to get a whole bunch of interesting uh pillage out of it and um cleaned up all the laptops afterward so now Jessica's going to talk to you about how we fix all these things all right cool thank you so yeah at a high level I just want to mention that we were coming in sort of retroactively trying to fix this and it's a lot better if you can Implement some of these fixes proactively it's sort of the Green Field stage it's quite hard to go back and try to implement

them in sort of a messy package environment where for a while people have been just um publishing packages without say consistent naming conventions or restrictions or that sort of thing so it was a bit of a struggle it's it's taken a while but um you know here are some of the things which we've considered or uh actually implemented and and I'll talk a little bit more in depth about all of these so the first thing that we did when we realized we were sort of vulnerable to rce here was registering placeholder packages so the idea here is that you register a package with the same name as your private package uh in the public repository so

then you're sitting on the name so an attacker can't also publish a malicious package with that same name because you own it so the pros here are that it was quick for us to implement all we had to do was you know write a script to identify um the internal packages that were potentially vulnerable and then we wrote another script that would automatically register packages with those same names onto the public npm so pretty quick easy to do um and then another benefit here is that uh it was something that the security team could do by ourselves didn't require waiting for CI infra or developers to change anything on their ends um but it did

have uh some downsides as well uh namely it does not prevent recurrence and what I mean by that is if you run the script to register packages uh today and then tomorrow a developer goes and publishes a private package uh you're going to be vulnerable to dependency confusion for that package uh until you run the script again to register those packages so you would need to run it on some kind of schedule and then in that window between runs of that uh script you would be vulnerable uh it does also leak your internal package names of course because you're publishing uh those packages to the uh public package store so if you have any sort of sensitive data uh in

your private package names or for whatever reason you don't want to publicize that this wouldn't work for those packages we were also concerned about gaps in how we were identifying those private packages that we would need to register placeholder packages for uh namely what we were doing was scanning GitHub for package.json files and then you know running those through confused and then any uh private only packages we would register placeholders for and the with that is the GitHub uh code search API uh only indexes the default branch of a repository so main for instance and so if a developer uh has a dependency referenced in a non-default branch like a Dev Branch or a feature Branch or if they're uh

publishing a private depend dependency that isn't referenced in code at all we would miss it um we tried to patch this up by also looking uh in artifactory at our private package stores but again like um identifying the private packages is sort of a complex problem here and of course uh it's not really great uh internet citizenship to squat on package names in the public package repository you know there are questions there about like um should we be doing that like it's maybe a little bit rude um another thing that this uh does help with though is that developers can also specify internal sources for packages without going through ART Factory for instance by specifying something like a python

extra index URL uh in the install command and this is a great way for them to open themselves up to this vulnerability and very neatly evade all of the uh artifactory level defenses that I'm going to talk about in a minute which is great so uh the placeholder packages since they're not at the artifactory level um will still protect the developer if they do something like that so that's good and again this was like the the Quick Fix we did just writing a couple of scripts but you know as I mentioned we thought this was maybe sort of a dirty solution so we wanted to look for something better another thing we considered was requiring Scopes so this

is an npm specific thing npm allows you to put basically an organizational prefix on your packages so at Google at square whatever and only members of that organization can publish uh a package under that scope so the pros here it seems like something we probably should have been doing anyway simply from a developer best practice perspective it would make it a lot easier for us to identify internal packages so like I said on the previous slide identifying internal packages without this sort of consistency was a complex problem for us and it does not leak package names like the previous option because um it doesn't involve publishing uh public packages with the same names as our

private packages however it is an npm specific uh fix and we wanted ideally a language agnostic fix since as we mentioned this does potentially apply to multiple languages and we do use lots of different languages at our company it's difficult uh culturally and technically to gate developers from publishing packages so if developers are accustomed to being able to publish private packages uh they don't want to suddenly have to jump through hoops and simply asking them politely to do it is maybe not a reliable security control we would have to actually prevent them somehow and we weren't really sure how to do that um also since we were coming in sort of late on this fix it's hard to

retroactively go back and add Scopes to a bunch of package names it involves you know updating any place where the package gets referenced to add the scope to the name and all these things very hard and of course we would still be leaking the package names to npm itself because the package manager would still look for the package uh externally uh even though it wasn't published externally so ultimately uh we decided not to implement this at least as a security security control maybe it would be you know developer best practice another option we considered getting into some of the artifactory level uh INF a stuff is telling artifactory not to look EX c l four

package names which match essentially a Rex pattern so this is a feature called artifactory exclude rules so you would configure artifactory uh for instance if a package name starts with uh Square do not look for it externally only look for it internally so the pros here are that it is language agnostic uh it does not leak package names at all uh because artifactory will not even look externally for packages matching that pattern and again since it's at the uh artifactory level devs don't need to change anything about their behaviors or anything in their code the downsides here is that if you do not have a consistent naming pattern for your packages which again we didn't uh it's

actually very hard to generate these rules right if you have a consistent prefix it's easy uh but what we ended up having to do was again reuse that script where we identified internal packages and then uh generate exclude rules for exactly those package names so if package name equals equals Square Dev tools do not look externally and what Justin said we had about a thousand of those roughly and so we had an exclude rule for each of those um so again this runs into the recurrence problem you have to run the script regularly to identify uh and exclude those packages it runs into the package identification problem that I talked about before and of course if there are many such rules

there are concerns about slowing down package installation times now we didn't actually see uh very much impact on package installation when we did this but you can imagine that it might not be super scalable like eventually you would probably start bumping into that issue if you have enough uh rules so again we're looking for something a little bit better uh we did use this to some extent but like we think we can do a little bit better as well so another option that we used was artifactory priority resolution which is uh you simply tell artifactory to prioritize a particular source so for instance if a uh package exists on that priority Source it will not look for it

externally it will simply serve the uh priority package immediately so again doesn't matter what language a package is for it's at the artifactory level it's not leaking package names cuz it doesn't even look for them and again devs don't need to do anything um and it's sort of a nice elegant solution um it doesn't require actually identifying the the private packages per se just the fact that they're in a particular source is enough uh the downside here is that there is an edge case where um you do actually want the uh public version of a package when there is a public and private package with the same name and that edge case is the if say a developer

at some point published a fork of a popular public package like let's say a developer says Ah I think it would be great to publish a private Fork of uh react right 10 years ago and then they stopped maintaining it and now we really want to still uh have react the public version The Working version served when they go to install reacts we want the public version and not the private version and therefore if we simply turn this feature on it would break things because suddenly developers would start getting that old vestigial Fork that really doesn't do what they expect anymore and so what we had to start doing was identifying instances of those vestigial Forks of public packages

cleaning those up and sort of gradually turning on this feature as we cleaned things up and also there might be multiple internal sources and the question comes of like which to prioritize so so that was a question that we've been working on as well so as an overview here um the placeholder packages were something we quickly did sort of as a stop Gap um we decided not to go with requiring Scopes um we wrote some scripts to do exclude rules and we've been using exclude rules in combination with sort of gradually turning on priority resolution as we clean up those vestigial packages and one more thing I do also want to mention is that uh pinning dependency version

will also help with this um it's not really a dependency confusion specific fix and the complexities of whether or not you should pin your dependencies is uh probably worth another entire talk so I won't dive into it but just so you know that is another option to consider and with that um we will open the floor up to any questions and I'll also just pop over to this link here where you can see some of the sources that we use the tools and uh yeah thanks again for uh Alex for this great medium post about the issue confused from from vimc um artifactory team has done great work at this as well and thank you also

to uh bsides and the organizers and for you all for coming to the talk so thank you all right let's hear it for our speakers as per usual we will be taking questions through slid uh if you don't have access to it yet it is bides sf.org qna and that'll give you directions on how to connect and submit questions uh we have one question queued up uh first question why not use artifact reconfiguration that blocks packages with a prefix from public repositories it's definitely yours uh so I'm not sure I so I think this is talking about exclude rules so why not um have an exclude rule that would like um identify um public packages and block those um I mean you

can't rely on public packages having like a specific prefix right public packages could be named literally anything so I I don't think that would work unless I'm misunderstanding what you're proposing here so if I'm misunderstanding you feel free to come yell at me afterwards but yeah basically um there's not a consistent prefix to check for in public packages all right no other questions on uh deck yet we'll give the audience a half a minute to see if there's anything else uh question in the crowd um I'll take your question but I'll need to repeat for the mic so of that get into get into more attack scenarios like I don't know Ty of squatting or just

malicious yeah definitely this is oh sorry yeah sorry I need to repeat the question so uh your question was uh does this mitigate any other types of attacks like Ty ah has this triggered you guys to explore other vectors of attacks could be mitigated by this yeah great question so um I think while the attacks you mentioned are very similar um I think the motivation behind us doing this sort of attack was to push teams to sort of prioritize fixing it and since we didn't really have a clear uh path in mind for how to fix something like typo squatting or some of those other um vulnerabilities we did not go with um red teams targeting those however we did

actually end up doing a second red team on the same vulnerability um when we decided that uh developers were being too slow about fixing it after the first time we did it so all right we got another new question coming in uh any interesting General learnings from the red team exercise anything you'd avoid doing in the future you won this so again this was our very first one and the people who were doing it none of us really had a lot of red team experience before so it was very very much uh learning as we went um I did talk about like if you're going to if you're going to pop up a prompt anyway you might as

well ask for root in a situation like this so do that again um we definitely should have done more testing we had some other problems with our script where um well for one I already talked about the it falling off when you rebooted your machine uh but also we had some other plans for some other things we were other targets we were going to execute on and it turned out that we were never able to run there because we weren't able to test it well with that particular environment um so a little more testing definitely would have been better um I have one yeah go ahead yeah another thing is that uh when we actually got shell we hadn't really

planned for what to do next that much so we kind of felt like the the dog that caught the car and we're like oh yeah we'll run a secret scanner let's look at the secrets all let's jump and get some code in there I think if we have sort of planned for what to do after we got shell you know as they say shell is only the beginning um that would have helped with uh sort of demonstrating more impact as well all right next question um after prioritizing internal dependencies uh what caus you to not pick the highest version number or latest version public uh do do you find developers have that many internal packages with the same

name that's definitely all right so um oh you're talking about like combining priority resolution with always prioritizing the highest available version I think that could also cause some issues like there are situations where you don't necessarily want the absolute newest version of a package available and developers uh will say have some additional restrictions say in the dependency file where um I don't think we could just override that and and say you know here's the version that you should be delivered yeah all right uh that's all the time we have for questions today I'd like to thank again our speakers uh for coming today and giving this great talk uh i' also like to thank our sponsors socket

security who are providing these wonderful goodie bags that keep our speakers coming here because clearly that's the only reason you're here all right