Jarrod Overson - How did 8 million developers download an exploit with no one noticing?

all right good morning everybody my name is Brian I will be on stage as little as possible so that you can actually learn something today we're gonna learn about nodejs and why exploits can stay in the wild for weeks without anybody knowing it the person's going to tell you about that is Jared Overson Jared's a director of engineering at shape security he led the velvet of shapes enterprise defense he's a frequent speaker on modern web threats and cyber crime has been quoted by these little organizations like Forbes or The Wall Street Journal CNET you know fly-by-night stuff he's the co-author of O'Reilly developing web components what animals on the front of that book a carp we're really running

out of animals for O'Reilly books he also created Plato which is a static analysis tool for web applications and he frequently writes and records topics about reverse engineering automation everybody please welcome Jared hi everyone thank you very much for being patient it's always the fun part of this this is nailed down and so that I'm not gonna be bending the whole time I will hold onto it and hopefully it works so yeah I'm Jared and I'm we talking about an open-source supply chain attack that affected millions of developers late last November so to get started just to set the stage a little bit how many people feel uncomfortable looking at instructions like this good this is a

security conference it's good to see so many hands up at other conferences they're like what's wrong with that that's that's just normal you copy you that into your command line and you move on this should terrify you a little bit because you're just piping whatever you get from the internet into something that interprets it into commands that your computer runs I you should feel uncomfortable you should leap over the keyboard and stop your co-workers or at least let them know what's going on before they enter this in you might write blog posts about how much it really really bothers you and how much other people shouldn't be doing this and maybe if it really really

bothers you you'll catalogue a list of all the offenders just because that's what some people on the internet do but now how many people feel uncomfortable in doing this good this is a security conference in please you really should be putting your hands up because this is the exact same thing you're downloading code from someplace written by somebody you don't know through the same channels that we basically browse the internet all day and then running it on your computer and who knows what it's gonna do now this is not necessarily exclusive to npm a-- no this is basically a problem all the way around and it is a serious problem that has manifested into a real world damage

so late last year there was a malicious or there was an exploit of a very very popular NPM package those downloaded eight million times before it was even discovered and this is we're gonna be going over right now we're gonna be diving into the different payloads already talking about how it happened what it did and then finally where it leaves us but first of all why should you be listening to me at all and why should you not be coming to track a where there might be something that is being talked about by somebody you like better I'm Jarrod Orson director of shape security we protect a lot of websites against large-scale automated attacks I

write a lot about reverse engineering JavaScript breaking web application stuff like that was an old-school video game hacker way back in the day that's how I got interested in computers if you ever downloaded any Starcraft or fallout or total annihilation hacks things like that might have been some of my stuff way way long ago and no I won't tell you what my handle was you can find me at J's Overson just about everywhere and one of the things that I inevitably come across when I say that I work in shape is a bunch of blank stares because no one knows what the hell shape is shape security is company based in California like all companies are and

rather than actually talking about it I'd like to ask you first have you ever heard of YKK anyone know what YKK is there's a few hands that show up right there now I guarantee you every single one of you uses YKK daily because they make virtually all zippers that exist in the world if you take a look right now at your crotch and look on your zipper and you'll probably see YKK on it and if you don't then your bag or your hoodie or whatever else is everywhere anyway shape is the YKK the Internet security we use this virtually everywhere you don't know we exist we protect lots of login forms Ajax requests we're basically the reason why

you're starting to see fewer CAPTCHAs and have to login less super exciting stuff but we do well we do a ton of stuff with JavaScript and this is what what got me into a research position going through the reverse engineering of malicious JavaScript I and maintain a repository of all bad JavaScript that I find because I just find it really really funny so first off how it happened started with a package called event - stream actually how many people are nodi people here no D not newly very very few that's totally normal in web conferences where I talk the audience is always like things are insecure at security conferences everyone looks at me as like people use JavaScript yes

javascript is very very very very popular easily the most popular language that people develop it nowadays just because you have to you have to develop JavaScript that are in to target the web and no genius is a very very popular environment to program javascript in event a stream is a package that existed in the npm our ecosystem and yam is the node package manager like gem pip stuff like that Sipan if you're a pearly person if that stream was one of the first the first major modules that abstracted away a lot of the weirdness of nose streams streams are native objects that allow you to stream data and do all sorts of magic event stream made that easier now being

a early package that abstracts away something low-level in an in an ecosystem that expands rapidly you will get a lot of people to depend on you very very quickly as of this screenshot which was about a year ago now shortly after the event there were over 1600 dependents and now those are direct dependence not even second third or in tier dependents as soon as you spiral out from there you start to see that this package was dependent on by virtually everything it had 1.2 million weekly downloads and was just a very very popular package was maintained by a very very popular developer dominic Tarr this is not a no-name developer this is a guy that a

lot of people know a lot of people trust and he writes a lot of code he is almost 3000 followers on github and just publishes code non-stop one of those people we all see those people and they look like mythical code junkies that just constantly push out everything and the entire world builds software on their backs now this is this is someone that you should trust right when you're looking for packages you look for somebody who has a long history of trustworthy code because we're not doing

September 2018 dominic get ownership to a user called write nine control now why would somebody give a very very popular package to somebody they didn't know anyone do open source stuff here publish open source a lot of people probably want their packages to be used it would be exciting to have a package as dependent on as event stream a lot of people when they're working on something that are passionate about thank eighty one day you'll get a hundred users or a thousand years is 1.2 million weekly downloads is just astonishing why would somebody who has just give it away well luckily he told us in the issue that initially prompted a lot of discussion around this the user is asked

for you know and that's okay I mean these are open-source developers they're not getting paid they deliver something that is useful to the world and we can't expect them to develop it and maintain it for the end of time somebody else has to end up taking it over in order for it to continue and if somebody is interested in the package and they email you it and they look I guess not super-shady and even if that even crosses your mind then you just give it to them not exactly the best from a security perspective but it's just a natural way that these things are going to progress we can't really expect a whole lot more

when we're taking things for free and using them for free so right now in control I gain trust by committing several innocent changes this was a package that had been largely untouched for a couple years things like updating dependencies updating documentation adding some different examples all basic stuff normal stuff on September 9th 20 18 right 9 control added a new dependency in released version 3.3 six new dependency was flatmap stream anchored to a dependence to version 0.1 zero but let's talk about that care real quick people familiar with semantic versioning is that semantic versioning when I say that is that something is like yeah I know that really I don't know what the hell's going on I know

there were no hands for anything there how many people have no idea of semantic versioning is that's ok all right cool semantic versioning is a way of describing a a essentially a decreasing or increasing risk of a particular change so you have a major a minor in a patch version embedded in the entire version string patch version should be things like security updates bug fixes something that doesn't actually affect any behavior minor versions should be added features so if you if you update to a minor version and you're not you shouldn't expect any changes in behavior for the stuff you're already using a major version change is just a new API change something breaking incompatible

you bump them at you major version so this allows us to to to basically update to things with some understanding of the risk involved of course it's literally just a string that anyone can can change however they want there's nothing actually hard coding these rules so it's just up to the developer to manage them now NPM has the NPM package Jason which is where you'd find dependencies description stuff like that has a way to clamp these versions so that they will automatically update to a range that you are okay with so if you specify a carrot and you you lets a clamp it to 0.10 with a carrot then it matches all minor and patch versions so

if you wipe away your dependencies and reinstall you will automatically update anything that has updated the minor and patch version and if you use the tilde it only does the patch versions so it's a good way to make sure that you're staying reasonably up-to-date with your dependencies but this is a roomful of security professionals you probably already are getting a witness to what dangers that might entail so right now in control then removed flatmap stream and updated event - stream so they remove the flatmap stream dependency an updated event - stream - version 4.0 zero this doesn't look like anything strange nothing malicious has actually happened yet the right control added some features and remove those features and

then bump the major version nothing wildly strange about that the total time between the first commit in version 4.0 zero was 12 days nothing malicious has happened so far so on October 5th 2018 was 31 days after the transition of control flatmap stream at 0.1 1 was published this had malicious code in it now event - stream 336 I installed fresh will include new versions of flatmap stream because of that carrot event stream at 335 was stable for two plus years an NPM by default included dependencies with that carrot so basically any project that installed event stream by default will automatically pull in this new dependency automatically or will update to 336 next time event stream gets

updated which in turn will update flatmap stream to 0 1 1 so time between malicious control and discovery was 77 days the time between flatmap stream 0 1 1 the malicious package and exposure was 48 days so there was a malicious code in the wild being downloaded by developers all around the world for 48 days so next section here we're gonna go into what it did it was three payloads and we're gonna dive through each payload and show you how this this this malicious exploit dug its way into an actual mobile application that could have existed on your phone but first we're gonna talk about how it was actually discovered payload a the first payload used a

method that was deprecated in node version 11.04 first deprecated methods get deprecated in a documentation then the next major version of node deprecates them with runtime warnings and then never the major version after that removes them from the standard library so in version 10.00 this method was documentation only deprecation and then version 11.0 to warn about usage usage of these methods node version 11.00 was released only eighteen days and to the exploit so what this what this meant was that completely unrelated packages started seeing warnings thrown to the console because they were using this deprecated method this is an issue open on an unrelated repository this is a node bond which is a popular node monitoring

process it monitor scripts will reload them when they when they change will will restart them if they die something that should be completely unrelated to cryptography deprecated methods so once these run time what these warnings started popping up on the console it understandably concerned a few people and eventually people tracked it down to event - stream and a user called falling snow opened up an issue essentially asking WTF there's something clearly strange going on here why did you transfer controller why did you give access to this user what is this doing why did you just say why did you remove it without telling anybody what's going on so how was it discovered pure dumb luck

the attacker who delivered the code into flatmap stream used a method that was deprecated but was not issuing any warnings at the time of development and then when no just naturally updated its are throwing warnings and if that never happened I don't know when this would have been found let's dive into what this actually did this is this is what flatmap stream version 0 1 0 looks like this is the non malicious version and if you are somewhat a suit in the realm of development it should look a little strange for me by default because it's just a bunch of code smashed up together this is normal for the JavaScript ecosystem because of just that the

transpilers compilers bundlers whatever else it just smashes up code into what distributable artifact you're trying to build and then if you try to look at it you're just out of luck and that's what we have grown to live with very flat map stream version 0 1 1 I had this it was 382 bytes added to the end of it looks very very similar to the rest of the code nothing that should throw you off right away but this is the the code that was malicious and caused the all the panic so payload a we're gonna dive into that this is the bootstrap payload we're in the security world right now we know these things work the first payload is

the fun part because that is what gets your foot in the door that is what what allows you to continue the attack and then dig deeper this is what it looked like all prettified we're gonna smash that a little bit because there's stuff that we don't really need to care about the first thing that you should be focusing on right now is this line right here this is the first executed expression this is a require of something that looks like a hexadecimal string require in no jeaious is your import method so you require dependencies and what have your required will be import into your into your script and you can access things from there so it looks like it has a decimal

string because it is a hexadecimal string once you unhex that you see that is requiring a file called dot slash test slash data so it's looking for a JavaScript file in its in its dependency file structure take a look at that and it's just a bunch of hex strings this file existed only for flatmap stream version 0 ones 1 did not exist in other versions but it really wouldn't bother people for seeing it because it was in a test directory and I guess people just assumed test directories don't cause damage so when you unhex those values the first two values are very very large they unhex to something that some binary data the other stuff on taxes into stuff

that looks a little bit shady and this is one of the reasons why of course you extract these strings and you obfuscate them because if you were looking at a dependency update you started to see references to a yes or are ciphers or cryptography you would probably be a little bit concerned because this is a dependency that really shouldn't be doing anything at all with cryptography so after some replacement of those values and some cleanup of the original payload this is what we're looking at now so the first line that we're dealing with is just the requiring of the test data we're throwing that into a variable that I've renamed test data to a little bit more clear the next step

here is looking in our process environment for a variable called NPM underscore package underscore description and we're assigning that did a desk description variable so the this environment variable is automatically populated when NPM the package manager which can also run scripts runs one of its scripts it populates environment variables for all of its data in its description so what we're doing here is pulling from the environment something that comes from a package JSON file so the description field and a package out JSON file and then we are passing that to the cryptography library in order to create a decipher instance so what this means is that this description is our key to decrypt something the next

couple lines here we're taking the first value from the test data the first really really long string and we are decrypting that with that key that we got out of our environment next few lines here we're creating a new module with module two constructor method modules are modules there are individual units of code logic you can ignore the paths that's an odious thing we're compiling the text that we have decrypted into R as the contents of our module so what this means is that that first test data that line there is a JavaScript file that is being decrypted and loaded up as a new module and then what we're exporting from that module is the second value of our test data so

that that's second really really large string so these end up being our payloads B and C so now this this is an important part here because this is this is the key to actually going any for any further with this analysis what we have to do right now is figure out where this key comes from and I was doing a lot of this research on my own because it's fun and I like doing stuff like that I'm not gonna be a brute-forcing in a massive massive range of possible keys just because that's gonna take too long out of the hardware for it so we need to do is find good package Jason files this is an example package

JSON file which is just from a completely unrelated package just sound could be something I was using there in the research and this is what scripts look like as they're defined and packaged up Jason they just have a name and then essentially something you run on the command line now this is what we are looking for this is what is being run and this is the D this is the structure of the package adjacent is the structure that all the data in the environment gets populated with so we're looking for a package Jason that has a description that successfully decrypt our first that first value in the test data so what we we are at where we're at

so far we have a script that deep crypts and compiles a new JavaScript module the key for the decryption comes from a package Jason descriptions somewhere the encrypted JavaScript from there comes from test data and the compiled it module exports the second value from test data so what this means is that this this script will only serve its purpose if the code runs from an NPM script in a directory that has a package JSON that has a description a description value that successfully descript decrypts the test data so what this means for us is that we just need to start trolling through package jason files now anyone know about packages and files where would you find the biggest repository of

packages jason files it would be in the and luckily when we wanted to go through all the packages there we could just use and gam package called all the packages because there is literally something for everything on NPM and the funny part of this is that it of course depended on event stream so as you're going through this you just have to accept you are already screwed and a lot of people are already screwed with something like this so strategy here was to iterate through every single possible package that existed in the NPM registry try to decrypt test data first value in our test data since we know that it's supposed to be JavaScript and we also know that different keys can

successfully decrypt data it's just not going to get you the right data we're gonna take that decrypted data with the key that we have and then run it through a JavaScript parser to make sure that we're dealing with Javas valid JavaScript we don't do it by hand and then if successful we have a winner now this is the this was a long shot this was all I was gonna do if it didn't work then I just was was gonna have to deal with it because it's hard to just figure out what keys for random encrypted data are that's that's supposed to be the way it is so I'm not getting an updated here there we go so what we see right here is

in just under a couple minutes that was actually 20 seconds to go through the entire NPM registry was a little under a couple minutes we find an icky that actually successfully decrypt this data the description we were looking for it was a secure Bitcoin wallet so some package out there the description wallet was being targeted this ended up being copay the secured Bitcoin wallet a payload B this is what injects this is what gets run when we find out that we were running within the project that we want to be running in and then injects payload C payload B we can now decrypt because we have the key and we're not going to go through every single line

we're going to focus on a couple so this this particular line right here is testing the arguments that are being used to run this script and if they do not match a particular regular expression then we are going to bail so what this means further is that we are more isolating the effect of this malicious code to a very very specific environment the way NPM scripts run you type in and game space run script space and then the script you are trying to run so our arguments here the first argument is NPM the first element in RV is NPM second index one is run script third next two is the script - name or the scripting the regular expression

we're looking for right now is something that's that includes build and then a literal colon and then anything and then at - release now because copay is luckily open source we can actually find out specifically what it was targeting and the the regular expression would match these scripts in the co-pays package at Jason anyone familiar with Cordova or PhoneGap it's a used to be very very popular it was one of the original ways of using web technology on mobile and native environments so we'd use HTML CSS and JavaScript packaged it up into a webview and then you could deploy on App Store's or iOS or whatever else so this this build step build iOS release build Android - release build

desktop - release they were building HTML Javascript and CSS into a native application and that is what the application was so payload be also had this line in it right here this is a deep dependency a deep legitimate dependency of copay - - and this is not a malicious file but the payload be used this file because the developers knew that this file was being loaded in the context they wanted to be loaded in when the application was actually running and what they did was inject the next payload into that file so that when that file was was loaded in the application their malicious code would run so to recap here payload B doesn't do anything

unless it is run in copays build stage it decrypts payload see just like pay low B gets grabs the same key cuz the package and project is the same in injects payload see into a file used in copays mobile application and pay let's see is then executed in the mobile application itself on the user's mobile device payload C [Music] it looks like this and we're actually not going to go through it the final payload and as a lot of us know the final payloads are actually the least exciting of research like this because this is just what the attacker wanted to do at the start and needed to figure out a way into the system in order to get it

to actually run it's a lot of web code it's document local storage passing things with global variables essentially what it did harvest it private keys it's the targeted wallets with a certain amount of Bitcoin cache pass a lot of this data onto a third party server so where does this leave us now this is not a node or NPM specific problem this is a problem with any sort of open source registry of code that you install any which way the problem here is that JavaScript and NPM are incredibly popular so we see things like this faster and we see more of it but this is a problem that also exists in other open-source ecosystems and it's

something that we're probably not going to see any shortage of so the good news here is that once the issue was brought to light the community responded very very rapidly of course the payload had already been in the wild for a month and a half but that's a whole different issue because as soon as people found it they were very very motivated to fix it they developed tools that would analyze your system to determine whether or not you had any affected packages obviously NPM took down the malicious package issued security warnings the community responded very very rapidly bad news is that it still happened multiple times since and doesn't really show any any slowing down this happened actually just

a few months ago this was another cryptocurrency application that was exploited and this was exploited via another another package electron - negative - notifier electron is the more modern version of Cordova it's the it's the what's not it's not it's not developed by the same court over people but it's the same sort of spirit it's using web technology and ours to produce native applications so things like Visual Studio code slack Skype those all use electron so this was targeting one of the plugin for electron that allowed you to issue native notifications and of course was used to steal Bitcoin because that's what people do and somewhat recently last week there was some analysis done on the NPM

ecosystem and it found that you could just exploit 20 people and take over essentially half of all an NPM packages and projects and this is a dependency system that has grown in a sub optimal way and it is powering a massive massive part of a portion of our technology now this could have been much much worse luckily for us real people this this malicious code very very specifically targeted a specific system in a specific application now I have to imagine that the person who went through the effort to actually do this once they they found out what they had probably regretted that they only targeted co-pay because this was used by virtually everyone millions upon millions of developers

daily and you could use something like Visual Studio code and also have this problem so you don't even need to be a node.js developer or use NPM at all to have potentially been affected this this package is also used by as your CLI Azure is Microsoft's cloud offering so you can also imagine the types of people who might be using the command-line interface for a cloud offering like that internal to Microsoft certainly and very very highly privileged developers all around the world it was a massive massive widespread problem and this is probably going to get a lot worse I to be honest this is not something that that is going to easily go away it

requires us to re-evaluate the way our systems are working and unfortunately are things like Ruby Python node and and all existing stuff that it's already very very popular I'll of these decisions involve backwards and compatibility and changes that will will push push the the state of security forward but are going to alienate a lot of your fans and it becomes a marketing and cost versus value decision and I think a lot of us probably know where security sits and most people's cost versus value justifications now the developer of a node has actually shown interest in this particular problem Ryan doll created node a little over 10 years ago now the project has not been for several years but has started

developing and showcasing a spiritual successor to node called Dino which is a a experiment to elevate some of the problems in the node.js environment so that we can better handle problems like this and a lot of other problems and one of the things that it does well is isolate access to privileged API the same things like what we do on our mobile phone so you don't want Facebook to use GPS because you're a normal person you can disallow Facebook from using GPS because if you don't they will do it because they're horrible people sorry for all facebook developers and anyone who views this stream you're probably a good person but this is something that we

don't have in a lot of our development environments and we don't have a node and adding this sort of feature and having this module isolation and sandboxing and compartmentalization inevitably involves a lot of backwards compatibility problems can be very hard to wedge into the existing no ecosystem so what can you do personally to address this you're all seen people I'd love security because a lot of the stuff isn't actually all that new to you but worried about your dependencies for people this is all free NPM install everything but no dependencies come with risk think twice before adding dependencies make sure you're locking your dependency versions so you don't get surprised when things change from underneath you you do

need to audit when things change you make sure those changes are acceptable when in doubt don't add it this of course involves this of course is something you should apply to open source development for server-side applications but also for websites all those script tags you see that loads megabytes and megabytes of JavaScript and slows down your connections and computer only how do those if you really need to because all of this stuff is risk and risk is okay if the value is greater than the potential downsides but a lot of times we don't actually make that determination we just throw things and see where it takes us so gamble when the cost is low on the value aside thank

you very much

[Applause] this mic the audience my quirky Thank You Jared if you have questions we have about time for two or three good questions or possibly one extremely compelling one if you want to ask a question please come to the microphone if you are leaving the room please leave through the doors at the rear and I'll have tips for you if you're sticking around for the next talk so any questions please use the mic so you talked a little bit about the plans to make the language itself more resistant to this are the things that you want to see either NPM or github or the other package systems adopt in order to make this harder to do yes but there are also

problems with what I would like that I would hesitate to recommend one of the one of the serious problems that does exist is the disconnection between the original source code and the distributable so a lot of people of course host code on github but then publish whatever to NPM there's no actual link or assertion that what exists in a repository is and it ends up being what exists in the NPM registry so like this this example is a good one because what existed in NPM never actually existed in github so you had to actually get the distributable and and dig through the distributable to figure out what was actually in it tieing a source or source code repository

directly to its distribute more of a breadcrumb trail to figure out what's going on but that does that does potentially consolidate a lot power in a particular company like github if they start to control more of the distributable x' which is what they're doing now so it's good for some things don't know how it's gonna play out in the long term though so the description is non humour human-readable ki and why isn't that being detected and also when you decrypted checking of every package did you run a linter to see if the description was a non human readable key so the the description actually was human readable it was this it was the string a secure Bitcoin wallet so it was

using that as the key to decrypt it was using that as the create key to decrypt its payload can you repeat that a different way

yeah there's nothing really special about taking the description out of this this was just it could have been any value in that it could have been it could have been something else specific to the project it could have been something local to the final system there was nothing really specific or special about that that key it was just I guess probably the first and easiest way to do to do that

so it seemed like part of the key to this attack was the obfuscation right the multiple layers of obfuscation that they put in place I'm given that these are kind of open source projects is that something like a method that we could attack this from right like mounting a lot not allowing that level of obfuscation in code so there are there are companies that that do audit dependencies and one of the things that it they do is look for signs of obvious obfuscation so strings that are hexadecimal values the problem there is similar to the first question I answered is that there's a disconnect between where are these source code lies and and who ingests the distributable artifact

so there's no actual there's no actual connection there so even though source code does exist in github and is open it doesn't mean that I can't develop something completely arbitrary and then publish that as the next version in something in NPM as long as I have access to that repository and NPM and and I'm an author and NPM then I can publish anything it doesn't need to be connected at all to what exists in github so having some sort of connection there is important but that is something that I don't foresee happening in the near future because companies playing well with other companies who knows sometimes that works sometimes it doesn't familiar with NPM and yet the

degree to which it might mean I get what you're saying about the disconnect between the packaging and distribution and it's just kind of black listing things so I don't know to the extent I would have even caught something like this so NPM on it is a great way to identify whether or not you have any doubt for people who don't know NPM audit is another command that can be run in the NPM package manager that will check your dependencies against any known security vulnerabilities or any security issues and then give you a list of listen packages severity and those for you the problem with that though is of course they only detect what they know

and there is a lot of noise there just like with a lot of lenders that are too aggressive as soon as you get a whole bunch of warnings that you know aren't actually that important then you're just going to ignore everything and the fact is when you're just blanket auditing all of the dependencies in a project you're going to come up with some security vulnerabilities but some of them just might not be that big video hey there there's there are a lot of security vulnerabilities that just won't actually cause you a problem ever because you're never running the code that has a vulnerability in a scenario where the vulnerability can manifest so when you run into audits like that and

you see a lot of things like that then it becomes very very easy to just ignore everything right another round for sure thank you very much