Human or Machine? The Voight Kampff Test for Web Application Vulnerabilities

okay uh well everyone I'm super bummed that I'm not giving this presentation to you in Athens much rather be there than in my apartment but I am here located in San Francisco California my beachside apartment so I can't complain about that and I'm really excited to be talking to you all today about research that I've been conducting since I joined cobalt thought IO which is a pen testing as a service company out here in the Bay Area so a little bit about me replicant tendencies but probably organic I started my career in Washington DC where I was mostly focused on nation-state actors and doing a lot of work in the national security and cyber security policy arena concerning

apts and then last year I moved to San Francisco to work for cobalt which is a pentesting start-up in San Francisco and a lot of the work that I have been doing is has been around the value of vulnerabilities so that spans from bug bounty to con testing to sort of what dynamic scanners bring to the table I've been helping us sort of assess what that means for application security so my co researcher on this is one of our technical program managers Travis McCormack he is a badass data Wrangler used to work at Walmart doing pen Nessen for Walmart and has the osep so a little bit about this talk human our machine the Void contest for web application

vulnerabilities some of you may not know this I don't know how many of you increase have actually seen Blade Runner it's it's like a 1980s movie in which police officers are hunting down robots who masquerade or appear like humans in the Void contest is a is a task conducted by a corporation to figure out if if someone is a human or a machine so my question for you the question that this this research presents is why does method matter why does it matter what vulnerabilities are discovered by humans or machines because in the end of all is a vom no matter the method through which a vulnerability is discovered and reported for the purpose of security

that goal is the same if you're responsible for protecting software you want to find it remediated as many vulnerabilities as possible to reduce the likelihood of a breach or if you are a black hat hacker you actually want to increase the efficiency of identifying vulnerabilities to increase the likelihood of a breach we're actually seeing this in cybercrime you know we're seeing a lot of trends around automating crime right I mean even even things as simple as like malware as a service it's not propagating so we're seeing efficiency we're seeing competing efficiencies on both sides right between black hat and white hat hackers there is also a very legitimate business case for discerning discerning between human and

machine analyst actually predict the global application security market is gonna grow up to nine point six four billion dollars by the end of 2023 it's a compound annual growth rate of twenty four point nine five percent in industry that's nearly ten billion dollars imagine telling that to phone freak in the 1960s could face basically criminal charges for essentially identifying business logic flaws with telephone companies the market is rising to the demand and today there are hundreds of companies geared towards application security scanners that cost tens or even hundreds of thousands of dollars are competing against open source tools we'll talk a little bit about that later and the efficacy of some of these tools remain unclear freelancers are looking

for the same bugs that a specialized pentester working for consultancy is searching for results-oriented work is now pitted against time box work the question of human or machine is now a question of ascertaining value in a results-driven market it's not part of the strategy for choosing vendors allocating resources and determining the best use for info sex greatest scarcity there's time okay so let's dive in a little bit I chose web application vulnerabilities for a reason from the data I can find web apps continue to be the most tested for pen testing at least as you can see here more than 65 of our pen tests the Cobalt conducted last year were web apps or web apps and api's and what about bug

bounty bug bounty is the same everyone says bug bounty hunters predominantly hack web apps for bug route 90% of the targets were also web apps I would love to see some research on other targets like external networks or cloud configuration I think it's a great area for food for a future research project for someone but for the purpose of this talk I'm sticking to what's most valuable ok so a little bit about this research the scope of this research is dynamic scanning pitted against black box testing this includes advanced out-of-band scanning which finds bones like second order sequel or blind process scripting I'll go into that a little bit more detail later we wanted to answer three questions web

vulnerabilities can dynamic scanners find what are the vulnerabilities that only humans can find like dynamic scanners cannot reliably identify them and one of the vulnerabilities for where scanners will not automatically populate results but where automated tools can enhance efficiency to conduct for their exploitation we generated input from cobots core which is our pool of roughly 300 freelance vetted pen testers they had a lot of opinions and a lot of disagreements which is great and then for the examples I've pulled for this presentation I relied on public bug bounty reports ooofff examples and ports were there I would have loved to have pulled from cobalt results but we actually don't disclose customer vulnerabilities even if they're

anonymized and of course humans can have a little proxy as a treat I'll get more into that later okay so what kind of web application vulnerabilities has cobalt and finding for finding a lot of bones miss configuration continues to dominate it's a pretty broad category the ranges from cookie attributes to business logic bypass wasp List msconfig as number six in its top ten web app security risks but for the second year in a row miss configuration leads in our findings followed by a cross-site scripting attacks it's a little bit hard to see the data here so I'm just going to read it off for you it's miss configuration cross-site scripting authentication incessant sessions other sensitive data

exposure missing access controls insecure object reference components with known vulnerabilities across it cross-site request forgery redirects and forwards server side request forgery and remote code execution okay so here's where the Machine wins it's a pretty comprehensive if you see this dynamic scanners work as many of you know by injecting malicious payloads they test access points where they are communicating with the front-end scanners are programmed to understand arguments and function calls and they can't attack can detect vulnerabilities and headers verbs fragments and Dom they can also identify some miss configurations and find components with known vulnerabilities one of the disagreements among our pen testers was whether scanners could reliably find trickier bones like second order sequel

or block blind cross-site scripting as many of you know a standard process scripting or sequel attack for instance produces and medium results like an alert prompt that says hello world or data spilled into an input field when you see that you can immediately recognize the success of a payload but what happens when a payload is successfully injected but the vulnerabilities output isn't produced immediately blind vulnerabilities mean the request response is office gated in some way making it difficult or impossible to interpret and then out-of-band vulnerabilities means response does not return within the same interface through which the time with blind or out-of-band runner abilities you actually won't see the vulnerabilities output immediately this is often because an application

activates the payload at a later point thus requiring the pen tester to have back-end knowledge of the system to know if the payload was successful there's no echo in the cave are you yelling loud enough or is it actually a tunnel I've concluded that for the purpose of this research the answer is yes machines are awarded blind second order or otherwise out-of-band vulnerabilities you might be asking okay why the asterisks it's there for reasons Stanners scanners still require manual setup and produce a significant number of false positives so a human will have to configure and ease any scanner and sift the results I mean also as many of you know from running your own enterprise scanners like they

produce a ton of results and you actually have to go through them and figure out what's positive you know what's a false positive what's a true positive and then also be able to figure out what's a priority for you right so even if a machine can actually report certain vulnerabilities it's still gonna require human to go through and actually figure out what the hell's going on and then here's just like a couple of examples of some open source scanners out there there could be an entire talk on the efficacy of certain scanners over others some companies claim their tools can find a certain vulnerability some other companies might dispute those claims with their competitors I'm not

gonna say some of these companies are lying but let's just say the results can prove inconsistent yeah but maybe a scanner could find a certain type of vulnerability but with a slim success rate or a high false positive rate you know with a ton of configuration maybe a scanner could find a certain type of all you know it depends so I highly recommend checking out shake hands research comparing the price and features of webapp scanners on the market both commercial and open source and you can ping me at any time and I don't provide that for you okay so here's where things get fun we debated this among the COBOL core and as I said

earlier there was not complete consensus but the three vulnerability types that we generally agreed upon that only humans could find we're business logic bypass race conditions and chained exploits so I'm gonna provide you with some cool examples just to show you exactly what happens with these kind of vulnerabilities and what you know to give you a better understanding of why humans excel at finding these okay so the first is business logic bypass I just pulled this from bug Bonnie report it's a really great example a few years ago a handful of corp aunt asters worked on a bug binding for uber what they found was a pretty interesting business logic bypass that led to an idea whare

so how do they do it first they figured out that any user could create a driver account but that account couldn't be activated until uber verify their driver documents you couldn't access the driver you couldn't access the uber driver out until you were verified or at least that's what Oberer intended when looking at the requests screenshot it on the left you can see that there's a parameter called allow not activated that was set to false so what happens if you set it to true while they actually obtained a balance session token' navona in the response there was a field called is activated set to false when you change that to true they were able to access

the driver out this later allowed them to access another driver's name license plate last trip UUID last passenger name number of passengers origins and trip destination through an idea waar as my colleague Travis likes to say building sounds hard it's way easier to break things in the midst of buildings business logic attacks exploit design flaws or unanticipated abuse of business logic like having a user controllable parameter that has trusted business logic bypass requires thinking about an overall system about overall system architecture architecture its contextual one of the things that I really learned in the last years they're doing this research and my own experience doing what about pad testing it's how important the fundamentals are

a lot of this is actually is about understanding the principles behind how about location is functioning understanding the architecture understanding I'm exactly what programmers were deciding when they when they built this application on so it really requires taking about three thousand foot view of the application and also diving in and seeing exactly what's wrong so the other one that we agreed on was of race conditions first of all I love to handle cash money it's like an amazing name second of all I could paraphrases Vaughn for you but Shopify actually explained it really well in their bug bounty report so I'm just gonna copy exactly what they said Cash Money reported it was possible to

bypass the e-mail verification process and a partner's dashboard doing so would have allowed a partner to request access to a store under an email address a partner do not own if the store had a staff account associated with that email address the staff account would have been automatically converted to a collaborator account and added to the partners dashboard without any merchant interaction we tracked down the bug to a race condition in the logic for changing and verifying email addresses we fix it by locking the database record during these actions and requiring store administrators to approve all collaborate and requests so in my video this is also another clear example of understanding business logic as well to

really excel at finding bones like this you have to understand the intended processes driving the business and ask yourself yeah but what if I try this instead for cash money the answer to this question was fifteen thousand two hundred fifty dollars for a time of check time abuse race condition honestly my opinion pretty well deserved as I said the Shopify bug was exploited through a time of chop time of use race condition as you see here this is multiple step process to successfully exploiting this kind of vulnerability race conditions are not caused by walk are caused by not locking the file meaning there was a race between when a file is opened and not locked by the

process it was bad with Shopify you can probably imagine how even more serious a race condition is for Bank like when approving and transfer I am sure that some of you are sitting on your hands right now believing these scanners can find race conditions some of you probably think that the machine deserves to win this one Canas can find this actually yes a scanner can but it has to be source source code review or static analysis that would probably be able to find this race conditions are probably found through fuzzing tool as well but that's not the same thing as a scanner and fuzzing isn't necessarily a function of a vulnerability scanner it doesn't actually tell you what the

vulnerability is you have to find what breaks first through fuzzing so basically I picked you both I took you out a little bit scanners can find theoretically fun race conditions but not dynamic scanners not like a huge amount of configuration or like a very specialized tool and it would probably need to be some sort of static analysis of fine stuff or you need to use fuzzing to find a race condition and in that case I don't qualify fuzzing as a scanner because it doesn't identify the vulnerability okay last up is chained exploits I think this is self-explanatory so I'm not going to get too in-depth here on the example I included is actually one from one of our

core pen testers I'm just gonna read off the hacker one some word for you Shopify Shopify infrastructure is isolated into subsets of infrastructure this bug bounty hunter reported it was possible to gain root access to any container and one particular subset by exploiting a server site request forgery bug and the screen showing functionality of Shopify exchange within an hour of receiving the report we disabled the vulnerable service began auditing applications and all subsets and remediating across our infrastructure the vulnerable sub-site did not include chocolate core after auditing all services we fix the bug by deploying a metadata concealment proxy to disable access to metadata information we also disabled access to internal IPS on all infrastructure

subsets we awarded this $25,000 as a Shopify core RCE since several locations in the subset do not have sound Shopify core data in systems or excuse me do have access to some job by core data in systems so if you can find a scanner they can successfully find an RC II through service ever court request forgery I'm all yours in the meantime I think we can safely say only a human can find this and I also just want to say for the record that chain exploits really proved that even if you find a low level low level vulnerability you know some simple cross-site scripting for example those can be used right to get to a more severe vulnerability and

so even if it's something that's relatively minor you know it's always better to remediate it I mean a classic example um you know I reported four bug bounty company recently when I was doing some hunting for them you know something really simple as user enumeration you know that I found it's a really low level thing like it was just on you know that they didn't report the right error message but even those minor things can lead to a successful brute-force attempt right which is one of the one of the fastest and most common ways to hack into a web application so I seriously recommend you know you take every report you find seriously and you know you do

take a serious look at even the smaller or lower lower risk vulnerabilities have been presented to you because you just never know what can end up being chained so the truth is the void contest is just not right upon testing humans and machines rely on each other automated tools and scanners are intended to drive efficiency and enable human creativity but working together humans and tools are powerful classic example this is you know authorization flaws that I do are you know dom-based cross-site scripting RCE session management file upload bugs another one recently that I've been looking into is I'm sure many of you know of a subdomain takeover right I mean you can't automate the out of

that pretty easily and be able to find some domains you know you don't need to go through and click on every single webpage that you find scripts are available for you to do that and it's just up to you to figure out exactly how to execute so just a case study for you to see this is I do are I just pull this from burp burp suite or port swagger you know as an example of how to find this you set up bird proxy you send request to intruders you pick the position you find a section of URL the refers to an object head over to a payload tailor the number and start the attack and then you use the results to

take other actions like a numerating accounts right so go through and you see 200 200 200 great gives you a sense of what's going on you know then you can move from there so basically the tactical takeaway for this like what is what's a time to go take away tool price does not equal the quality of the results so you know even if you are basically something like a wasp zap for instance is a tremendous and powerful tool that you can use it's free it's a dynamic scanner that it also does you know crawling for you as well to help you find and do recon on web apps you know there are a lot of really great

open source tools out there that do a highly effective job of finding vulnerabilities so I would not I would not necessarily conflate price with quality and the number of vulnerabilities that the reporter does not necessarily mean the quality of the results - as I said earlier with chain exploits you know you do need to be aware of lower level bones that are reported because something as simple as like a minor you know reflective process scripting can potentially do more serious damage if it's chained and then of course creativity if creativity is greater than narrow thinking then you're gonna find better runner abilities um you're gonna you know maybe get more money if you're a bug bounty hunter I

mean you're gonna help more fun and when you're looking to hire a pen tester you know it's the same thing you you want someone who is able to think outside the box and be able to think strategically and creatively it also has a really good sense of you know business logic and the fundamentals and so if you can take anything away from from this talk it's that humans and machines work together and finding vulnerabilities and we have more fun that way so thank you so much for joining me today like I said I'm sorry that I couldn't be in Athens with all of you wonderful people but hopefully we will unite another time so yeah thank you all so much