BG - Everything I know about prototype pollutions: how to react when confronted to a brand new vuln

hello everyone welcome to bci las vegas back in person um i just say a few words uh first of all i want to say uh thanks to our sponsors uh diamond sponsor lastpass and palo alto networks as well as our gold sponsors i just named three here amazon envision and blue cat a little bit of housekeeping please silence your cell phones the talk is being recorded it'll be available on youtube we also streaming live so uh you know make it make it nice here if we have questions at the end make sure that you come closer to the podium we don't have microphone in the room so the presenter can hear you repeat the question for the stream and

answer it um picture policy you should know by now besides a strict policy for taking pictures make sure that you have explicit content from everybody in the frame should you take pictures keep your mask at all the time before i introduce brad here vladimir turkey how many people can speak french okay so we'll do it in english but it's just because it's you so vladimir is going to talk about a certain form of pollution from what i understand so hey you have the flow mercy dimitri uh but thanks for coming to my talk i hope you will enjoy it there will be a lot of javascript inside but don't worry we will cover everything feel free to sign very widely if i'm not

speaking properly in the microphone vladimir you everybody calls me vlad i'm a staff engineer at datadog i'm also a node.js core collaborator i've been working on application security for the last six years in a startup that got acquired by datadog last year so we are building application security products feel free to reach out also feel free to follow me on twitter at pauldefette today i will take a chronological approach on that talk so there will be a lot of context stuff technical stuff a bit of uh self-soul-searching uh it will be kind of a mix of a lot of things don't worry the technical part is still the biggest part and a few disclaimers there's a lot of storytelling that's a

very personal presentation it will cover a lot of things i've done as an open source contributor not as a data dog no screen employee and there will be a lot of back and forth about what i had in mind back then and you will understand why i say that now also last disclaimer i've built some commercial content online trainings about javascript security including prototype pollution and i found it was fair to to predict the disclaimers okay so it all started uh with the backbone t program so how many of you are familiar with npm as a package ecosystem okay a fair amount of people so just quick reminders it's the most popular package ecosystem in the world

and you can take this graph at any time in the past few years it will look the same npm as a javascript package hosting repository is the most growing and the most popular package repository in the world and that's been so for almost a decade now there are millions of packages on npm and npm serves a huge variety of content of course they're javascript backhand packages node modules but they're also javascript front-end modules like people get react bootstrap from npm so front-end libraries are also downloaded from npm you have webassembly code that is distributed over npm because it's used on the web so wait no why not and you have native executable uh those are the

native add-ons used with node.js so if you want to for instance use sqlite you're likely to use the sqlite3 native addon on node.js and that might be pre-compiled or not for instance data.org as a couple pre-compiled native executable on npm to patch node.js directly or even code in cc plus plus rest that can be compiled at install time for node.js so why why do i say that well in 2016 the question of vulnerable slash medicine's package was a bit undefined npm did not really npm as a commercial entity did not want to do any strong efforts in that direction they had a lot of other things to do and that was fair from their point

of view um sneak existed but was still fair new and that's that's a commercial tool and the node security project which was basically an alternative to sneak was deprecating because they got acquired by npm to do something totally different than vulnerable package management so the node security project came to the node foundation and say hey we've got this database of vulnerable and malicious packages do you want it do you want to be in charge of npm security as the node.js foundation and we said yes we were probably young and crazy so we created a node.js ecosystem bugbunty program and hacker1 was extremely nice they offered us a free plan for the ecosystem and we also have

a free plan for node core so if you find minorities in node.js feel free to report them on hacker1 i will be happy to review them uh and we started the backbone t without rewards and depending on the case sometimes the owners of the vulnerable packages would give rewards so for instance a versailles company had vulnerabilities in a package themselves and they rewarded the hackers with 500 dollars from time to time which was very generous and that was a lot of work handling this burgmanty program especially when i was by my own for the first six months because we had hundreds and of reports and usually reports there are trends when people find a vulnerability

in the in search ecosystem people will do everything they can to find the same vulnerabilities in similar packages and that's actually important for for the rest of the talk and then in 2018 january we get a super report because it impacts 12 packages on the apm ecosystem comes from a security researcher from canada named olivier and there are 12 impacted modules but the weird part is it talks about prototype pollution and we had no idea what prototype pollution was back then that was the first time we heard about it all together and we did not find any literature so the summary was utilities function in the listed modules can be tricked into modifying the prototype of object when

the attackers control part of the structure passed to this function this can let blah blah blah how can you override methods in the javascript standard library and that's where we start to go technical and we forget about history for for a bit so in javascript you've got basic tips types you've got booleans objects numbers big ends strings symbols and undefined and pretty much everything else is objects so dicts are objects strange objects are objects regular expressions are objects null is a specific object and objects are objects and map and sets all this higher level data structures they are objects too and objects have method so for instance if i call undedict foo 1 has on property

foo it will return true and hazard property is actually a property available it's a method available on that object uh if i check has on property and has own property it's false and hazon property tells you if this object owns this property or if it's come from somewhere else so now we know that we can call a method named has on property but it does not exist on the object itself so how is it available where does it come from well let's use a debugger because the debugger is probably the most marvelous piece of software in the history of software yeah i'm terrible when i'm asked to mentor juniors i for them to learn the

debugger rather than putting prints everywhere um i never force anyone to do any anybody ah anybody i'm doing anything don't worry so if we debug it we realize that there's actually a property named prototype and that has on property is actually attached to this prototype so how does that work in javascript objects can't can have a prototype and when a method or property basically methods are properties in javascript is not found on an object we look up on its prototype so if you want to access a property on an object and if not available on the object directly we take the object's prototype and we check on that and the prototype is actually an object itself

but this prototype might have a prototype so if the prototype does not have the property you're looking for well by recursion you look on it on the prototype of the prototype or on the prototype of the prototype of the prototype until there's no prototype available and that's why we talk a lot about prototype chain in javascript so let's talk about prototype chain uh on the left hand side we define functions uh constructors and on the right hand side we use them so let's create an object that's called myproto that should be a prototype for certain objects and it has a function a property named foo that is a function then we define a class with the old

style in javascript where you define just the constructor and you attach a prototype property on that constructor and here we put our own prototype with bar and hello world as properties and then we defined a new style class that extends the old style class and as bars as a property so if we create an item an object from this class here well it has the property hello available because it's available on the prototype of old class old styles class that is actually implemented in our new class so that's just basic inheritance in any object-oriented language but with prototypes and if we call baz it's available because it's on the prototype here and if we call bar it's available because

it's on the prototype here same thing if we call two it will share the same prototypes and if we create item three and we call the method set prototype off because why it's why not we can this method exists and we use my proto well now the prototype for item three will be my proto and we have selected an arbitrary prototype and what's interesting is that prototypes are objects and they are not defined multiple times in the heap there's only one instance of the prototype as an object in the hip so for instance item 2 and item 1 you know they are created with new on the new style class so their prototype is the same object we

can check equality object equality on their prototype value but this prototype has another prototype that is the one from the old style class because inheritance because the new class is actually extending the old style class and all of these prototypes they actually have another higher level prototype that is the object.prototype the default prototype for objects in a javascript heap and on the other side item three it's prototype it's my proto and the prototype of its prototype is object.prototype okay so the gotchas are objective prototype properties come from the prototype or the object when they are not found on the object we look upon them on the prototype and prototypes are the same for two different objects multiple

objects can have the same prototype that are exactly the same object in hip and we can access the prototype from the object directly that's where javascript is a bit special so here i've got a class and i create two instances of this class one with b sides and the other vegas and if we call my item temba that has on property show prop it will tell us false also it's worth noticing that hazon property is actually available on the object prototype it's coming from the higher highest prototype in the prototype chain okay so we can call object dot get prototype off and that will return the prototype so in our case show prop is accessible on the prototype of my

item i don't think anyone has said that much prototyping talk in in few years then you can also access the prototype by calling underscore underscore proto underscore underscore that's a kind of outdated way to do it it's deprecated officially but we can't really remove it from the web platform whether that's the front end or the back end because it will break everything so there are options to disable it i will see that later but you can access the prototype through the underscore underscore proto underscore underscore property or each object in javascript has a reference to its constructor and remember the prototype is just a property on the constructor so if you access the constructor of an object

then the prototype value on the constructor you have access to the prototype too and just to re-explain that prototypes are single instance of objects if i take my item and my item too and i compare their prototypes they are exactly the same objects okay i think i'm good with explaining your prototypes work in javascript hopefully it was not too confusing so what's a pollution now that we know what a prototype is let's pollute them a prototype pollution happens when an arbitrary payload handled by the javascript codebase can overwrite properties or methods on the prototype chain and that's pretty much it and that happens usually when a merge function is called we'll see why now so here i'm using the

library in an outdated version it was part of the original report by olivier on prototype pollution so i create a malicious payload with a property name underscore underscore proto underscore underscore remember that's one of the way of accessing prototypes from an object and then i place a property named oops on it with it works i create an object named a and i don't mutate this object at all anymore because on the next line i say before and i check if a has a property named oops either an a is on the prototype chain and it will say undefined because there is no oops in the prototype chain or on a then i call hook.merge on a totally

brand new object nothing to do with a and json.pass my malicious payload i will explain why we use json and pass right after that and then i recheck if a has a oops property and actually tells us it works because the call to the merge function actually mutated something in the prototype chain because when the merge function was called it checked if it could merge the proto property of this object and it did so it accessed the proto property the prototype of our brand new object and added a property name oops on this prototype but this prototype is object.prototype the global prototype for all objects so basically all our objects in this javascript group hit

no real return it works when you call oops except if they have their own property name oops why do we use json.pass here because if you define the object directly well it will already have a proto property and it will be kinda ignored by the engine but there's a gotcha in the historical implementation of json.pass that makes this proto valid only if it comes outside of json.pass i won't go in the depth of the implementation of that but the gotcha is that for this exploit to work it must go through json.pass but so why then as i told you let's check the merge method and the merge method is actually recursive it calls merge again on itself

because you want to merge objects recursively because you want to merge nested and sub-objects of your object and that's where the prototype pollution happens because once again you will check if the object that is the target right now the first one here has underscore underscore proto underscore underscore as a property this will return true so you will merge property on the prototype because recursiveness i hope that's not too confused okay other example in that case we don't need the json.pass version because remember all objects in javascript has access to their constructor so their own prototype through the constructor so there's another exploit there used to be another exploit in lodash where you would do basically the same so

here we've got constructor prototype is that mean true we create a b object before we check b dot is mean it returns undefined we merge the payload on a totally different object and when we call b that is that mean it will return true because we polluted the prototype chain again and why does it matter well lowdash is actually one of the most popular javascript package on earth it's been downloaded more than 200 million times in april that's the downloads per month and it's been downloading 8 billion times since the package was created it's pretty much in most javascript code bases and it has been having multiple prototype pollutions meaning that the world javascript ecosystem was at risk

with them okay let's take a break from the javascript and saying prototype prototype and prototype of prototype and recursion and recursion and recursion and go back to the report to go a bit slower so we started to discuss on hacker one and the report is still private for a lot of reason even four years later but he's the gist of the discussion first thing we did we acknowledge having the report and we just gathered as the bugbunty manager team and be like okay what do we do with that now is that really a security issue or is that a quality issue is that a widespread defect in the javascript ecosystem or is it something that actually can be used

to impact applications on their security posture and discussions went on and on and at the same time i started to reach out to nathan la forest nathan la forest here with the main maintainer of hook the first library i showed today and i told him hey nothing i i don't understand how it works under the hood what i explained in the previous slide i did not understand back then just reading a report and it was hard to understand so we we sat together over the internet and started to put debug points and try to understand how how how that worked and that's how we understood prototype pollution and it blew our mind and we know

that there is a systemic bug in a lot of javascript code base based on that uh we know that everything that's merged or deep clone with recursion might be vulnerable to that as long as it takes arbitrary payloads in input is it a security issue that that's a one billion dollar question maybe i'm a bit exaggerating but not that much so we challenged a bit and the reporter actually provided an exploit in an existing code base node.js code base some kind of cms blog management i won't name it because it's the reason why the report is not public yet because it's been fixed in it but never published properly um and they actually highlighted a crash in the application so node.js

applications are writing in javascript and javascript is by definition single threaded like worker threads are an exception to that but no js applications they are thing they are usually single threaded and all http requests are handled by the same thread at the same time with a lot of asynchronicity so if you crash one thread in a node.js application well you crash the whole process because there's only one thread that's why error management is really critical and if you manage to find a crasher you're not crashing one request you're crashing the web server itself so they found a crash by replacing a method by a string which is fair and the codebase was calling this method and that was

crashing because it was a string at this point um so we found the denial of service because that's that's the definition of a denial of service you crash to applicati application server and we also found cwe 471 a modification of assumed immutable data that was a good candidate back then with our knowledge so based on that we had okay we have an exploit at least one real life exploits that has an acceptable outcome has been acceptable a malicious outcome uh and we found some kind of reference in referential we found this cwe saying hey this has been acknowledged as a group of vulnerabilities by the industry so at this point we are mostly thinking yeah that's probably your proper

vulnerability let's accept it as a vulnerability report so a bit of housekeeping we asked olivier to publish 12 reports instead of one single one because we had to communicate with library maintainers for them to fix and we didn't want them to know the list of other vulnerable packages uh we reached out to the maintainers shared the reports and we started publishing fix and publishing reports and we had like a few cwc cve going out in 2018 and what happens after 2018. well that's a good question yeah since 2018 there have been 193 cv published for prototype pollution 2022 is not over um and not all projects attribute cves for instance if i'm not mistaken sneak does

not apply uh published cvs on a lot of packages when they're not popular enough and the hacker one project for node ecosystem for npm ecosystem has been closed meaning there might be hundreds of other occurrences of prototype pollutions in the wild i kinda remember stats why it was one of the most the fastest publishing cv machines in terms of attack classes there have been one sce in chibana we'll go on that later and in december it was a repo there was a report about one cve on pass server we'll get to that a bit later too and more recently last month the royal institute of technology in sweden published published a really cool paper on the topic

the reference will be at the end of the of the talk it's a good write so i told you let's talk about an se because we found dos but what about sce over prototype pollution okay so as i told you node.js is single threaded so if you want to do tasks that are cpu intensive you will be blo you will be monopolizing the single thread and node.js has been designed for ios for asynchronicity so if you block the main thread you block the wall up and you kinda does it temporarily so kibana use a hack when it has to do some computing it start a child process which is very fair until we had worker threads in node.js

two years ago that was the only way to go and node.js shares the environment of the parent process to the children one and this is defined as a javascript object i think you kind of see where this is going right and node.js has an option that's called that's an environment variable called node options and basically that enables you to pass command lines argument to node.js but through an environment variable so instead of doing node.js dash something then your executable you can do node option your flags then load your executable which is something handy well what happened in kibana is that someone found a prototype pollution to actually write the node option property on the prototype chain

so when the child process was starting it was inheriting the environment from the parents including the node option newly defined and their environment viable on the prototype chain and they would be able to call dash e for eval and run arbitrary javascript code from the command line through a prototype pollution and since you're running arbitrary node.js code you can call another child process with a shell in it you can call the file system you've got an sce directly on the server so that was pretty bad that's been fixed and i know that the kibana team has been doing a tremendous job on that the past vulnerability is more recent the report is from december but it has been published only this year

if i'm not mistaken past was a standalone startup that got acquired by one of the major companies of the valley i'll let you google where that is but it's not google and they deprecated the project and the current version of pass is the open source community version of that and that's basically a rest api in front of mongodb it's used to build backends for mobile applications and basically it's it's an api it's a web api over mongodb so you can write objects you can read objects all of that and it's vulnerable to prototype pollution because it's called the merge function at some point that's that he exploits but also it uses a library named b

bison.js which is the default library used by the mongodb driver in node.js so the attacker can upload a document on mongodb through pass that's basically the point of this piece of software you can upload upload and retrieve documents in mongodb so good and those documents are formatted in bison and bison actually allows you to register functions that will be executed when you deserialize code from mongodb but this is by default disabled you need to enable the eval functions property on the bison library to make the code executable but there's a prototype pollution so you can pollute eval function for all objects in the hip and when bison is checking oh is eval function to set to true yes it will be

because of the pollution and because eval function is not default defined on the object in bason library because if the basin library had a real default version not a per default default property meaning that if the option object was updated with is this property existing on the object yes no if no its value is false then it will not be vulnerable but what they do is just does this property exist on the object or its prototype chain because it's easy to do and because of the prototype pollution you could do that so basically the outcome run arbitrary javascript code on the node.js server once again you can call any node.js module fork start spawn a shell do whatever you want

that's that's your sce and that's a terrible slide ordering for me i'm so sorry let's go back to the cubana one for one sec oh we think that node does not allow the dash e in node options anymore so you can't evolve keyboard as fixed you can check the report sorry about that how to prevent well remember that our merge function in herck here is the fix basically we just needed to prevent the property proto to be taken in account and that was good enough because at the only vulnerable path to herc herc was not vulnerable to the prototype dot constructor.prototype path so there are other ways when you call an object in javascript when you call a property it's a good

practice to actually check if the property exists on the object directly that's why we have has on property and you can make sure that the property is actually not inherited and there's an alternative you can actually create objects that won't have a prototype at all so null despite being an object does not have a prototype and you've got a method called object.create that takes a prototype as an argument and create a new object with the argument as a prototype so let's take our example again and here i create a with object that creates null so we run the attack and a is actually not vulnerable to this attack because it just doesn't have a prototype so there's

nothing to pollute but there's there's a drawback is that i can't call has on property or the usual methods of the objects in javascript because they're generated from the object prototype i think one of the main way to prevent prototype pollution and that's actually pretty straightforward is data validation and sanitization just to make sure that the objects that are coming from the user that are by definition interested don't contain properties you're not expecting so there's a very very very cool library named joy for that and you define the shape of the object even the size of the integers on it regex to validate properties whatever you want it's a very good library to do proper type and

validation on objects in javascript and by default it will prevent objects from having and and unplanned properties so shape your object and if properties are not expected they will be marked as invalid which is pretty cool sidenote object decentralization is timeless in javascript it's mostly json.pass and that opens the door to a lot of other issues over object injections one of the most impacting one is nosql injections you you can check my previous stroke just google my name and mongodb and you'll see a few talks on the topic if you're interested so that that's the conclusion now what do i think now what do i think four years later almost 200 published vulnerabilities later did we open the pandora box or did we do

the right thing well during the process of accepting the first prototype pollution report there was a lot of uncertainty i think we were good because we involved other people we trusted including community leaders as maintainers so we did not stay as a close mind team especially important because in that triage team under burgbunti i think i was the only one working full time no lyranto there was only two of us working full-time on application security and the other one were just node.js developers with no professional background in security especially so we did i think we did the right thing by gathering uh group intelligence on that uh i think we were fair in expecting an

example of exploit saying hey okay that brand new that makes sense can you demonstrate that it can have a real life impact on an application that would help us a lot and try to link to as little as much literature we can even if it was just finding the right cwe um i think we missed a few opportunities under process 2 because we had a brand new vulnerability class we knew it was largely impacting javascript as an ecosystem and we did not do what we should have done which is sit together with the node.js core team and say hey we need to compute the threat model and evaluate if there are something in node that could be

abused over prototype pollution and maybe we would have found the kibana case one year before it was published by saying hey look you can redefine environment variables maybe and i think i was really bad on communication i let olivier gave the talks on that which is fair he was the one who found the vulnerabilities uh fun thing uh the next year at uh west california i was giving a talk about node.js security model and it was in the next room giving a talk about prototype pollution um and olivier moved to something else after because he's a security researcher and we probably should have uh run that bike as community leader and security leaders in the javascript ecosystem evangelize

that's actually my first talk ever on prototype pollution and i think that's a missed opportunity on my side for the fun story a couple years ago someone tried to tell me there's a prototype pollution in my code and i demonstrate that it was unexploitable but sometimes you know um conclusion uh what now monitor incoming objects don't let proto or constructors come inside your hip from outside objects there's an option in node.js to disable proto altogether use at your own risk it might break some of your libraries but a good a good thing with same grip will help you know if you're if your code base or their dependencies actually use underscore underscore proto underscore underscore i'm saying i'm mentioning

some rep not only because you're in the room and enforce sanitization and prototype less objects a few links the two last ones are actually my classes if you want to deep dive into that the one that's here pollute me it's actually a github repo where i have written a vulnerable code base you can download it start the docker repo and try to exploit it i did that in 10 minutes yesterday so maybe it's not very good feel free to provide feedback uh reach out on twitter if you want the slides because i always forget to upload them between jet lag and busy conference time and the two first links are our report the first one uh i think it's the

yeah it's the kth paper and the second one is actually uh the sc in kibana i think i have time for some questions now yes

so the question is did we do any research to see if other languages were very variable to a similar class of attacks i'm not aware of that and i was actually reading the page about prototype oriented programming this morning it was like oh there's at least a few of the prototype oriented languages that's probably worth checking i think depending on the way inheritance is implemented in other languages it might make sense on the top of my head i would consider that language the trend on the jvm won't be vulnerable for that but that probably needs to be checked

you can't yeah the application [Music]

so the question is is there a go-to payload uh a cheat sheet to find them and because of those uh i i don't know if there is a list yet i actually think this one that replace has on property by null and same thing on that one uh that should do the trick and just check the object prototype on ndn write a script and you can use one single payload to override all of these values so you will soon see i think you've got a 70 chance of crashing the application if you manage to override them so yeah i i can try to upload something on github later

why was the back bounty shutdown for node ecosystem not not core mostly because npm was acquired by github and that was the best move for the javascript ecosystem and no guitar also npm started to do bug bounty i mean vulnerability report as a feature in npm within pm edit then got acquired by github that has security features so it did not make sense to have like an extra an extra initiative on that especially since we were doing that on our free time and people like github are paid to do that so they will do it 10 times better than we do

uh do we have uh regression testing on this vulnerable project um i i don't know but that's the magic about um about fashion in the bugbunty world is that so when we started the ecosystem but bunty the first report we get was a directory traversal in uh application used to serve static content and if you check the outcome of this bug boonty i think the probably the first 50 reports were that then when proper pollution was published we had probably 50 reports for prototype pollution so i would say that people who are getting trained to get karma and hacker1 are the regression testing for that if i may

um if i rephrase does googling search the code base now

um i i i don't think we will online we've been using that at all i was actually discussing with two people in this room earlier today about the fact that this kind of attacks are really hard to find a stat with static analysis if you read the kth paper one of the point is that they had a lot of false positives if i remember properly don't misquote me on that they had a lot of false positive and study static analysis so i don't have a i don't think scanning works yet for this class of attacks okay if there are no more questions feel free to follow me on twitter ping me provide feedback i'm looking for

them too and ask for the slides and thanks so much for being an amazing crowd [Applause]