BSidesSF 2020 - The Voight-Kampff Test for Discovering Vulnerabilities (Vanessa Sauter)

hi okay so my name is Vanessa Sauter I'm super excited to be here today I come from cobalt IO we are pen testing as a service startup based a few blocks away today we are going to build the voight-komp tests for web application vulnerabilities does anyone know what that is or am I just like super nerdy do we know okay okay cool it's Blade Runner theme so I apologies this my apologies this is a very nerdy talk and I'm a very nerdy person alright so a little bit about me and got started let's see this is gonna work one second yeah sorry guys yeah yeah alright cool so as I said my name is Vanessa I don't

actually know if I'm replicant or organic the Tyrell corporation has not tested me yet but I'm gonna take a safe bet that I'm probably organic I used to work at Brookings and the Aspen Institute my focus was cybersecurity policy within the parameters of national security a lot of the work that I did was focused on the tribe motivations behind apts and from that I I looked at the value of vulnerabilities and thinking about how vulnerabilities intersect with the way that apts operate and now I'm a cobalt I moved to San Francisco about six months ago and I am super excited about the work I'm doing right now my co researcher on this talk he's not

here today he's faced in Virginia is Travis McCormack he is a technical program manager a cobalt what I would call a Shepherd of our pen test and our pen testers he's a badass data Wrangler and a former lead specialist of security testing at Walmart and helped me a lot mostly with the data for this talk all right let's kick it off why does method matter in other words who the hell cares if a machine or human finds the vulnerabilities of all is a vomb right does it really matter in the end the method through which a vulnerability is reported so I think what we actually need to ask a two-part question the question of human or machine goes further than this

I recently finished a book called of the dead cow and what author Jomon encapsulate so well is the curiosity and joy of breaking things it is part of the human condition some of you in this room may viscerally understand the high from finding an exploit and find the light in the unexpected path it takes you others like myself may aspire to one day achieve that kind of rush this talk in part is a recognition of that joy it is a celebration of human curiosity and perseverance okay but on a much more pragmatic level there is also a very legitimate business case for discerning between human and machine analysts predict the global application security market will reach up to nine point six

four billion dollars by the end of 2023 that's a compound annual growth rate of twenty five point nine five percent in industry that's nearly ten billion dollars that's [ __ ] huge and the market is kept rising to the demand today there are hundreds of companies geared toward application security scanners that cost tens or even hundreds of thousands of dollars are competing against open source tools the efficacy of some of these tools it remains a little bit unclear we'll get a little bit more into that later freelancers are looking for the same bugs that a specialized pentester working for a consultancy is searching for results-oriented work is now pitted against time box work the question of

human and machine is now a question of ascertaining value in a results-driven market it's now part of the strategy for choosing vendors allocating resources and determining the best use for info sex greatest scarcity which is time so let's dive in I chose web application vulnerabilities for a reason from the data I can find web apps continue to be the most tested across the board as you can see here for coal more than 65% of our pen test conducted last year were web apps for web apps and api's okay so that's Cobalt but what about bug bounty it's the same hacker one says bug bounty hunters predominantly hack web apps for bug crowd 90% of the targets were also web

apps I would love to see some research on other targets like internal or external networks or cloud but for the purpose of this talk we're sticking to web apps I think it could be a great potential future research project for me or anyone else to think about other types of applications cloud or external internal networks but for right now it's just web apps okay a little bit about this research the scope of this research is dynamic scanning and blackbox testing this includes and this is really important this includes out-of-band scanning which finds Vons like second-order or blonde ii or sequel or blind cross-site scripting I'm gonna get into that a little bit more later so we

wanted to answer three questions for this research one won't foe know but what vulnerabilities can dynamic scanners find - what are the vulnerabilities that only humans can find meaning dynamic scanners cannot reliably find them three what are the vulnerabilities for which scanners will not automatically populate results but where automated tools can enhance efficiency to conduct further exploitation so we generated input from cobalt score which is our pool of roughly 300 freelance vetted pen testers full caveat they had a lot of opinions about this and there was a ton of disagreement which is great it definitely generated some conversation and then the examples I pulled I relied on public bug bounty reports Oh wasp and ports Witter

the researcher in me would have loved to pull from cobots results but we actually don't disclose customer vulnerabilities even if they're anonymous so we relied on public reports which are actually really great for this purpose and of course humans can have a little proxy as a treat okay so what kind of web app vulnerabilities is Cobell Friday first off we're finding a lot msconfig continues to dominate it's a pretty broad category the ranges from cookie attributes to business logic bypass wasp with lists msconfig as number six in its top ten web app security risks but for the second year in a row miss configuration leads in our findings followed by cross-site scripting attacks my apologies for the labeling it's a

little bit hard to see so I'm just gonna clarify for all of you top is Miss configuration followed by a cross-site scripting authentication in sessions other sensitive data exposure missing access controls and secure object reference components with known vulnerabilities redirects and Ford's server-side request forgery and finally our favorite RCE okay that's it let's dive in so here's where the machine went you can take a look at this list I'm not gonna read it out for you I'm gonna dive a little bit into why the machine wins here dynamic scanners work by injecting malicious payloads they test access points when they are communicating with a front-end scanners are programmed to understand arguments and function calls so they can

detect Vaughan's and headers verbs and fragments they can also identify some miss configurations and they can find components with known vulnerabilities one of the disagreements we've had with our pen testers and with people i've talked to as a whole is whether scanners could reliably find trickier vulnerabilities like second-order sequel or blind cross-site scripting a standard cross-site scripting or sequel attack for instance produces an immediate result like an alert prompt that says hello world or data spilled into an input field when you see that you can immediately recognize the success of a payload but what happens when a payload is successfully injected but the vulnerabilities output isn't produced immediately blind vulnerabilities mean the request response is obfuscated in

some way making it difficult or impossible to interpret out-of-band vulnerabilities means a response does not return within the same interface through which the attack was sent with blind or out honor abilities you won't see the vulnerabilities output immediately I think many people here assume at that point that the machine would not win but this is often because the application activates the payload at a later point thus requiring the pen tester to have back-end knowledge of the system to know if the payload was successful if there's no echo in the cave are you yelling loud enough or is it actually a tunnel I've concluded that for the purpose of this research the answer is yes machines are

awarded blind second-order or otherwise out-of-band vulnerabilities that can be used with a certain type of dynamic scanner or out-of-band scanner okay so I'm sure you're wondering why the asterisk scanners still require manual setup and they produce a significant number of vulnerabilities so a human will still have to configure any scanner and sit the results but for the purpose of finding vulnerabilities the list that I provided is an exhausted display of what scanners can find relatively easily but of course don't be fooled we all know who's pulling the strings I really hope it's not the Tyrell corporation okay there can be an entire talk on the efficacy of certain scanners over others and there's actually a lot of great

research out there I just included a couple lists of just like open source scanners that you can use that are actually really reliable some companies claim their tools can find certain vulnerabilities and then others might dispute those claims I'm not saying some of these companies are lying but I will say that the results can prove inconsistent yes maybe a scanner could find a certain type of vulnerability but with a high false positive rate or a slim success rate or with a ton of configuration like hours upon hours of configuration maybe a scanner could find a certain type of vaughan this is where it gets a little bit tricky here and I'm gonna get into a

little bit more detail why later in another slide I highly run I highly recommend checking out Shea Chan's research comparing the price and features of webapp scanners on the market both commercial and open source there's actually some really great research being done on this um I want to pause right now and ask what vulnerabilities do you think humans went up you can shout it out business logic okay any other phone in the back what was it access control okay anyone else what sequel injection okay one more social triggering doesn't count because its Web Apps but I like that thought yes so chained exploits basically complex well okay anyone else idea war all right okay we've got we've

got a pretty we've got a quorum of ideas here okay here's the list of vulnerabilities among our pen testers that we could agree only humans can find business logic bypass race conditions and chained exploits like I said there was not consensus so we what we had to do was boil it down to basically the three that everyone could agree on that was this list so now I'm gonna pull some cool examples just to show how these exploits work so a couple excuse me a few years ago a handful of core pastors worked on a bug bounty for uber what they found was a pretty interesting business logic bypass that then someone mentioned idor then then led to an idea

waar okay so how did they do it first they figured out that any user could create a driver account but that account could it be activated until uber verified their driver documents that seems reasonable right like you want your uber driver to actually have a driver's license you couldn't access app until you were verified or at least that's what you were intended when looking at the request screenshot on the left there was a parameter called allow not activated that was set to false so what happens if you set it to true they obtained a valid session token no wanna and the response there was a field called is activated it was set to false but when they changed it to true they

were able to access the driver app this later allowed them to access another driver's name license plate last trip UUID last passenger name number of passengers origins and trip destination through an ID of our okay as my colleague Travis likes to say building stuff is hard it's really friggin hard it's way easier to break things and to build things so business logic attacks exploit design flaws or unintended abuse of business logic like having a user control of a parameter that is trusted I pulled this research here from Marco Mirana how to prevent business flaws vulnerabilities in web applications and it gives you a list of why write weak enforcement of workflows poor parameter validation like we saw msconfig with

access control some of you mentioned that as well authentication flaws so really you have to have a very strong understanding of the systems and how they operate to be able to exploit this machine does not have the context that's required to be able to identify these kinds of vulnerabilities okay what's up next first of all I love to handle cash money that's like brilliant in my mind second of all I could paraphrase this phone for you but Shopify explained it really well so I'm just gonna pull straight from the bug bounty report cash money reported it was possible to bypass the e-mail verification process in our partners dashboard doing so would have allowed a partner to request access to a store

under an email address a partner did not own if the store had a staff account associated with that email address the staff account would have been automatically converted to a collaborator account and added to the partners dashboard without any merchant interaction we tracked down the buck to a race condition and the logic for changing and verifying email addresses we fix it by locking the database record during those actions and requiring store administrators to approve all collaborator requests in my opinion this is actually another clear example of understanding business logic as well to really excel at finding bones like this you have to understand the intended processes driving the business and ask yourself but what if I tried this

instead for cash money the answer to this question was fifteen thousand two hundred fifty dollars for a time of check time of use race condition honestly in my opinion that was pretty well deserved okay so what's up with race conditions the Shopify bug was exploited through a time of check time of use race condition as you see here there's a multi-step process to successfully exploit this kind of vulnerability race conditions are caused by not locking a file meaning there's a race between winnow files open and not locked by the process it was bad with Shopify and you can probably imagine how even more serious of race condition would be for Bank like when approving a bank transfer I am sure that

some of you in this room right now are sitting on your hands believing that scanners can find race conditions and that's because I've tricked you can a scanner find this maybe but if it's a source code review static analysis could probably ensure that you are locking your files but a dynamic scanner is not gonna find this and remember as I said it's dynamic scanning versus versus blackbox testing yes race conditions are probably found through fuzzing tools but that's not the same thing as a scanner fuzzing isn't necessarily a function of a vulnerability scanner and it does not tell you what the vulnerability is you have to find what breaks first through fuzzing another classic example of this

is buffer overflows you can set up fuzzers to run all night cause crashes but you're still gonna have to figure out what caused the crash you need to know what to look for you have to triage identify the vulnerability and the same thing goes for race conditions okay last up as you mentioned chain exploits I think this is pretty self-explanatory so I'm not gonna get too in-depth here the example I included is actually from one of our core pad testers and I'm just gonna read off the hacker one summary Shopify infrastructure is isolated in two subsets of infrastructure 0 acts ACB I have no idea how he actually wants to pronounce that reported it was possible

to gain root access to any container in one particular subset by exploiting a server site request forgery bug in the screenshotting function of a Shopify exchange within an hour of receiving the report we disabled the vulnerable service began auditing applications in all subsets and remediating across all our infrastructure the vulnerable subset did not include Shopify core after auditing all services we fix the bug by deploying a metadata concealment proxy to disable access and metadata information we also disabled access to internal IPS on all infrastructure subsets we awarded this $25,000 as a Shopify core RCE since some applications in this subset do have access to some Shopify core data and systems if you can find a scanner

they can successfully find an RC e through a server side request forgery I'm all ears like I want to invest money in that but in the meantime I think we can safely say only a human can find this ok how many of you actually disagree with this I'm very curious or have opinions I see a hand in the back that's good I like that ok the truth is if you do if you disagree with these results I really don't blame you it's not actually black or white it's not actually avoid contests you can't actually pick machines against humans as we discussed with race conditions tools are instrumental now in finding vulnerabilities just as scanners rely on

humans for configuration and sifting through false positives humans rely on machines to automate more menial tasks no one I think most people do not want to type out every variation of a JavaScript payload known to man to prompt a little one in an alert box that just sounds absolutely awful I'm very glad that scanners can help us with that so here's a new section humans and machine a new frontier a list of vulnerability said scanners cannot reliably find but humans can't really find without some sort of tool that they use authorization flaws xxe sam'l certain types of cross-site scripting insecure deserialization RCE session management and file upload bugs that was actually one that a lot of our pant

Astor's thought humans should be awarded but I still think the machines kind of got that one as well so we placed it in the middle so I'm just gonna give you a case study on why it is that some of these rely on both humans and machines and that is I do our I've just pulled this straight from ports worker so ports burgers instructions for fighting an idea are clearly demonstrate exactly how humans rely on tools to be able to find certain types of vulnerabilities first you have to set up burp proxy then you have to send a request to intruder find the section of the URL to refers to an object head over to the payload tailor

the number and then start the attack from there you can take other actions like a numerating accounts the machine it does most of the work but the human must first identify input it's a direct object reference and then sent a thousand iterations boom proxy does the rest of the work one of the things I realized in conducting this research is how valuable it is to know the fundamentals and when I say fundamentals I truly mean it I'm talking about simple networking protocols and web development understanding tcp/ip understanding HTTP knowing the basics of web development you know who I think would excel at finding a business logic bypass or a race condition someone who has like

multiple years as a sysadmin for my own perspective as a pentesting white belt it's helped me reprioritize how I approach application security and even how I study and test on my own time so enough of me talking I don't want to do questions I want to know what open it up for debate we have a microphone here basically tell me what you're thinking tell me what you disagree with tell me what you agree with this is not intended to be an authoritative list I want this to generate conversation it's an iterative process so with that let's open it up yes so it's only dynamic scanning it does not look at source code yeah and that's specifically for web

apps and web apps and api's huh um it was a question about the type of scanner that's being used and whether source and whether it was evaluating source code yeah does anyone think the machine does not win in this case and any of these cases what does anyone have like a Python script that they wrote that's on github or they like feel pretty confident that they can find a race condition because some people have said that yes it's using a lot of bug brownies okay so Tonya said that her old partner built a tool a race condition tool that's actually used for bug bounty programs cool anyone else have disagreements or ideas or agreements oh

yeah

yeah yeah agreed my question for you though is it is it looking at is it looking at the source code or is it doing dynamic a black box okay okay I would love to take a look at that so I will definitely check it out and see thank you for flying okay and I think that's it we're done so thank you so much everyone for coming [Applause]