Dmitry Zagadsky - Don’t end up with a pencil: Tips for shopping for pen tests.

thank you uh thank you to besides boston for having me um and thank you everyone here for uh attending um i'm kind of bummed that we can't be in person um and that i can't really see everyone's reactions but that's okay because at least one of us can uh pretend that everyone else laughed at my jokes so anyway let's uh let's start this and as a warning there's a big picture of me in the next screen so it might break your computer screen sorry uh if i can hit the next button hey there we go uh so who am i um i'm dmitry zagatski uh you can trust me because i have a cissp uh not really i mean i do but you can

trust me because uh i'm passionate about what i talk about and i i do a lot of research and i'm not afraid to be admit when i'm wrong so currently i am the avp of i.t security uh at uh potato credit union in rhode island i also do a lot of stuff with uh with dc 401 i participate in splunk boston user group i do the boston security meetup i haven't really done a lot of talking at conferences uh kind of i've been stuck with um thinking no one would care what the heck i want to talk about uh because i'm not like this big uh it's like a security researcher or whatever but this topic it kind of um stumbled on

and uh i'm really passionate about and i want to share some of the experience that i've had um one stupid fact i guess is that this picture is the only selfie i've ever taken to myself uh it was uh submission for the first layer eight conference uh the ocean ctf um and i've also been described as a person with a personal vendetta against penetration testers which uh is totally not true i'm actually kind of a pentest fanboy i think that pen testing is awesome and i wish i had the skills to to take over computer systems as well as the next guy but uh i'm stuck here defending them so anyway let's uh let's go on

so um what are we going to be talking about in this presentation uh there's a few things that that we're going to be doing uh first we're going to go over a quick baseline of you know what is a pen test and what isn't a pen test um how to pick a company that's good fit for you there's a lot of a lot of different vendors out there and a lot of different companies and what i've seen is that it really matters that you get someone that's a good fit it might not be the you know it's this uh there's a bunch of different ways to uh to get someone there uh things to look for in contracts

contracts are the are what kind of just runs everything and so there's a few things you really want to make sure that you uh look out for and that you have included um also when you're done uh obviously the thing you're looking for is a report that says what happened so there's a lot of things uh you should look out for and um you know that you that you want to do with all the reporting mostly this is a kind of a collection of things that i wish i knew when i first started there's not really a lot of resources available for the the customer side or the blue team side on the pen tests

so um a lot of the a lot of the things on pen testing are from the perspective of hey i want to be a pen tester or i want to know how to you know own a window system uh phillip wiley did an awesome presentation earlier on you know how to become a pen tester uh so this is this is a perspective from the from the other side um also a warning there's uh going to be some minor spoilers from the for the phoenix project um so uh hopefully uh mostly i i hope everyone's kind of read it because it's an awesome book it really uh really changed my perspective on a lot of things uh and it's

at this point like seven or eight years old so i think it's safe for spoilers um if there's anything else you want spoiled everyone dies in romeo and juliet so you know sorry um what this isn't is i'm not trying to bash pen testers or companies uh some of the things presented here might make some uh some people look uh not the best and that's not really my my goal uh i really want i really want to provide people with information again how how you can get a good fit for your organization and um not trying you know trying to show like what what could happen but not necessarily that it's always the case also i'm not going to be naming any

names uh so this is a lot of the stuff is uh covered under mdas and other things but again this is trying to i try to uh take uh some experiences of mine and others and and put a positive spin and more generalized for um you know a kind of a broader audience also unfortunately this is not going to be a comprehensive guide on how to do everything uh it's probably going to take a lot more time than an hour uh to do that but excuse me um it's we're gonna try to get it through and i'll be available for questions uh at the end uh and even though i'm on discord so yeah we'll see how it goes okay

so uh what is a pen test uh this is a collection of pens and they all squiggle and and work really nicely a while ago on twitter i asked uh you know what what is a pen test and i kind of asked for the wrong answers just for jokes um and the problem is that it's it's the kind of thing that you you know it when you see it but there's a lot of different interpretations uh as to as to what really constitutes a pen test and i feel like there really shouldn't be and it's kind of a problem in the industry uh when i first when i first started shopping for pen tests in in some roles like i thought i knew

what a pen test was and i thought that everyone else knew what a pen test was and i kind of you know some of the the things were lulled into a a fault sense of security um so here's here's a few things uh here's the original sandbox escapers uh there's pigs testing their pen uh not a pen test here's penn gillette writing on a pen uh not a pen test although someone said that teller talking to pen might have been and then there's this uh here's a tweet from tinker uh you should follow him on twitter he's awesome dude and this kind of illustrates the the problem it's getting more serious so the most common wrong answer

that people would give for what a pen test is is just you know it's just a vulnerability report with you know a few extra things tacked on and it really isn't uh it it shouldn't be and um you know this this kind of this kind of uh mentality uh is still pretty prevalent so here's a lot of words and uh this is probably the best explanation uh that i could find for what actually is a pen test and so this is in the nest 800-115 publication the technical guide to cyber security testing and auditing i believe so the the highlight is mine but it's you know it's a test that mimics real real world attacks and identifying

ways to certain commence security features in a system or a network usually you would involve you know launching real attacks on real systems and the most important part is that a pen test consists of uh combining vulnerabilities in one or more systems that you can use to then chain and gain more access than you could achieve just through a single vulnerability or next point um a lot of things so the and then the the things at the bottom are uh you know uh are you know some of the met some of the reasons that you would want to get one pretty much at the end of the day you get a pen test not necessarily because your compliance

program says you need to but that's usually a driving factor you get a pen test to verify your security program's effectiveness it's you know it's the the dry run of what everything you know everything you've done uh to to set everything up and that you know you're getting you're testing against real real style attacks hopefully uh against a real person who can who can put together things and maybe see some aspects of your systems that you didn't realize and and really get you some meaningful insight on how you can improve your security posture overall so there's a few different kinds of excuse me uh we're gonna go over a few different kinds uh just to set the baseline for what uh

what's involved so uh there's the external pen test which is uh pretty much just everything that is visible externally from the internet [Music] that belongs to you this is really useful for testing uh public facing systems customer facing systems you know vpn gateways anything cloud-based things like that it's pretty much what can a person anywhere in the planet see uh next uh a more important i feel is an internal contest this is where an attacker would start inside of the perimeter which with the way the technology uh is going you know everyone's saying the perimeter is dead so we'll see how that what that means in a few years but pretty much uh you would start uh you would have the

have an attacker system inside of your network or inside of your cloud environment or or somewhere else it's really good to simulate uh you know someone clicking on a phishing link uh some kind of a malicious x uh malicious insider activity uh or or just some other exploit where you wanna just see what happens if something you know gets in uh forget about how good your perimeter defenses are let's see what a worst case scenario looks like typically you would you would start these in a from the perspective of a normal user um but if you feel uh if you feel spicy you can have things start from inside like a server segment or uh some other privileged security zone

to see you know just explore different other options for testing so there's also the different ways the different amount of knowledge that the attacker has gained knowledge is power so the first kind is a no knowledge test this is only available this is only giving the attackers or the pen test company um what's available if they're ocent or scanning uh it typically simulates uh you know just what's publicly accessible and again almost almost always this is with all of your defensive measures turned on this can also be known as a black box test and this is typically you know it's a good starting point uh sometimes you may you may want to restrict this to you know to kind of give the

attackers a real um a real hard work a real hard job to do um usually you would want to go uh with something a little bit better or not better but so the next kind is a full knowledge test this is where the uh the testers would have as much information as they need that could be the design of the controls the kind of systems are in use addresses of high value targets that they might want to try to pivot towards um and if you really want to get clever you can choose to allow allow the systems past certain defenses so you can test what happens if that defense fails or if they can uh somehow bypass it and you don't want

to have to go through the effort of actually having the the testers attempt to bypass because not every social security tool system process is 100 infallible uh it's really good to have uh it's really good to have an opportunity to see what happens when when certain things fail and you get real a good idea of your defense in depth um then the next is a partial knowledge test otherwise known as a gray box test this is realistically where most tests actually happen you get kind of a combination of you know of of the two previous methods however my personal favorite is a gated test that is where you start with no knowledge and then as the test

progresses you disclose additional information to see if you can get different results so for example you'd start with you know a no knowledge test and see how far an attacker can go and then you say okay well here's all the defenses you're actually hitting and you're being blocked by them okay so now that they know all of that information can they deduce a way around that they couldn't that uh through other you know other means this really kind of uh uh it it's it's good and it avoids having them spend too much time wait not wasting time but spending too much time on ocean when you know in a real world attack you know the determined uh a determined malicious

actor has as much time as they want and realistically in a pen test you've got uh there's a time bound so it's a good way to simulate a more sophisticated attacker so we're going to start with a little bit of a story here and with uh i figured the phoenix project would be a good launching point uh for for this because a lot of this it seems like a lot of stuff in the phoenix project if you've read it has a basis in truth and a lot of times a lot of the people i've talked to who've read it they kind of associate certain characters with people that they actually know in their uh in their day-to-day job

and life and things so for the people who uh might not have read it a quick uh a quick overview of the characters is um there is bill palmer who is the head of i.t and is typically the protagonist of the book uh there's john pesci uh who is the chief security officer and there's sarah moulton who is kind he's the head of sales i believe and she's kind of the antagonist um there's brent who is like a top-notch i.t engineer who kind of gets everything done and they're working on a large application that they call project phoenix so in the beginning uh in the beginning of the book the uh the previous cio and the and the

vp of it are um for for reasons that aren't disclosed are they're fired and that's how the the the story starts where bill palmer gets suddenly promoted into being the the director of i.t and he sort of befriends john the security officer keeps trying to give him uh you know put tell him all these things that are wrong with all the systems regarding security so john's been pretty much chomping at the bit to uh to do an in-depth pen test because he wasn't really allowed to prior to the the departure of the two previous uh people they always you know said that they didn't have issue they weren't really any issues they didn't need an internal

test and they didn't want to have to deal with any possible downtime so john thinks that now the bill's in charge um they have an opportunity to do some some real good so let's they say okay let's let's pen test project phoenix it's the new big thing um let's see what we can do uh so oh sorry my slides here uh auto border but that's okay so john says he found a great company he's read an article about them in a you know in an industry magazine pen testers r us um and you know they seem pretty good so uh bill agrees and then john's like well do we want to do anything else other than that we you know we've got

pretty much everything is our you know we can we can do whatever we want um so bill suggests can they test some of sarah's stuff uh sarah's been getting in bill's way a lot and he wants him to test the the in-store systems he feels that they're probably like the people there probably aren't managing it right and really wants to get sarah in trouble so that way she can get off his back okay so they got that down and john wants to schedule it for january february so some of you might have kind of figured from that part of the story that there's a few there's a few uh uh traps here waiting for them but

oh man okay sorry my this slide was supposed to auto only show the the part the okay never mind sorry okay so pretty much how do you do goal how do you set goals in scoping for uh your your test um it's your environment you you realistically should know where all the bodies are buried uh so if you don't know where you've got some weaknesses or gaps or if you have the ability to uh you know defend against a sophisticated attack you should probably work on that first um you really kind of want to have a good idea a good vulnerability management program going uh and everything else um the other thing is you need to determine what's

important to you and that's different for every organization um is it your data is it the system is the availability of your systems um is it web apps that take customer information or shopping carts or vpn portals is active directory do you have a lot of wi-fi access points for guest access or something like that uh it really kind of depends and again it's uh it's if you if you don't know what what you what's important and what deserves testing then you probably should start there uh the other thing is uh do you want to test in production are you going to test it in the test environment a qa environment dr um each of those things carry

different uh caveats so like if you're protesting in production great it's a really good test but if you break something you're breaking live production systems is that a risk you want to take um and then you know for the test in the qa and dr uh so they're you know they're not the live production systems but are you sure that they're configured the same way as production and then can you know can stand in for production uh and you know if you're getting audited for compliance does that does that meet the smell test kind of uh the other time is you may want to just say i want to test dr or a test system uh specifically

because they want to see if your you know if your controls there are just as good as on production uh because a lot of times you know dr controls aren't as good so why would someone attack production when they can just steal the backups and get done with it um the next thing is you want to see what are you what are you actually looking to measure um are you do you care about the technical controls so well your firewalls your ids's uh ips's wax all those different things yeah all the blinky lights you want to make sure that you're getting the bang for the buck do you want to make sure that your teams

have the ability to detect certain attacks do you want to make sure that everyone's adhering to a certain process so you know if they do detect it do they follow the playbook that you guys had set up you can have everything set up 100 go out well you get a detection and just staff just isn't well trained uh or you know it doesn't follow what what you expect you get weird results and a lot of times a pen test might kind of highlight those things next you want to include social engineering and physical bypass or something along those lines um as as you may have heard uh the weak link is really usually in in layer 8 wink wink uh

so it's uh a lot of you know it's it's important to do that as part of a test and you kind of sometimes you have to be a little bit um delicate with with how you scope it and what you what you are okay with you kind of have to determine what level of evil uh you're okay with performing uh on your own employees and then and then see see how that goes it's it's it's it's a delicate situation it's hard it's sometimes hard to uh to do uh the next thing is you do not want to try to make someone look bad or hang someone out to dry that is a bad idea even though it's

so like a lot of times you say oh you hear okay yeah we're gonna own this system uh and you know and and get domain admin credentials and it's gonna be great and yeah you know you can you think okay i'm going to stick my pen testers on this other people and and show how they're not actually doing their job the right way that is a bad idea if you that that just breeds problems um it might you know it might pass your mind but you can't have that be um realistically any any part of what uh of of what you're doing uh so after you go through this exercise you should end up with actually a bunch

of different scopes or a bunch of different goals for for different opportunities and what you want to do is you want to collect them all down and you know keep track of them um because realistically in a in a test where it's time-bound you don't really want to just say okay just do everything uh okay somebody can do everything and it'll take a month and a half or more and they'll charge you you know lots and lots and lots of money and that's not really you know you don't want that's not really an effective use of anyone's time so if you if you scope things properly you can you can test specific things and then

have some kind of you know room for discovery to say okay hey yeah we saw this thing that's really bad but it's not necessarily in scope do you want to go and um you know do we want to go and look after it and you should be open to saying yes go after it as a customer okay so for the security weekly listeners i hope you have your drinking glasses uh handy because we're about to mention some hot topics where do you start to look for vendors other than google you can google you know pen testers in my area or or whatever and pinterest is in my area just probably ends up on a dating site

but as silly as it sounds the pci approved scanned vendor list is not a bad place to start many pci vendors offer pen testing services because pci compliance requires that you do pen testing uh you know it's it's a place to start also financial auditors um it's again i was a little bit surprised when i when i first ran across this but it's true uh financial audit firms um do pen tests the only thing is is you you want to be careful about the appearance of a conflict of interest or or any kind of uh in incorrect separation of duties um because the um you know if they've got the same people who are auditing your financials auditing your

it controls and then doing the pen test that verifies the ip controls that might be too much in one you know too many eggs in one basket for um for like a regulator or a compliance audit or anything like that doesn't mean it's bad but you want to just take that into account next is uh event sponsors you guys are all at an infosec event and oftentimes companies that do excellent pen testing sponsor infosec events i hear there might be one that sponsors this one but i'm not going to name names uh so you know it's a good idea opportunity especially kind of like not really have any pressure uh if we ever do anything in person you

know go to go to a booth at an event say you know ask some of the questions that we that we might go over here and get a you know get a quick feel for for how the company operates um you know and it's one way to to find them uh also word of mouth uh everything everything is great with word of mouth uh recommendations from from colleagues friends other people in the industry uh really carry a lot of weight uh the problem is is that you just need to make sure that every company is different so if you um you know something that's a good fit for someone else might not necessarily be a good fit for

you but it's a good place to start so once you're talking to a vendor what should you ask uh there's a bunch of stuff but first thing you want to do is pretty much you want to determine how they operate how what's their um what's their methodology um you know you can ask a question like you know what describe a typical engagement to me uh you know how do you start where does it go from there they should be going over thing uh you know different phases of the engagement like recon vulnerability identification exploitation uh what they do for reporting uh things like that um if the test is gonna tran you know uh if the test is gonna go over multiple

days how often do you get status updates uh do you get a uh you know a dedicated person uh to to deal with things like that um another thing is uh what are the qualifications that the testers have so this is honestly a thing that i have significant internal conflict with because as a person who does not have a college degree and who only recently started caring about certifications you kind of want to have you want to get an idea as to what you know what the what the people who are going to be operating in your environment um uh have i guess as far as uh qualifications certifications like uh you know ceh or uh os uh ocsp

um those are a good uh you know good to haves um the other thing is um you know the kind of people like get an idea as to what kind of people they have like are they all um you know are they all nsa agents or uh you know do they you know do they do teaching or what have you um you know obviously having a diverse group of people is is always going to help um uh in an environment and especially because you'll get uh you know if you get a p if you get a give people who have a white background um you'll be able to have a more effective test then so this one you might not get an

answer on if you if you ask this but uh i i feel like it's worth asking um do do the staff do presentations at conferences or do they do podcast appearances do they have blog posts uh anything like that so i would say it's not a requirement but it kind of gives you an idea about the person and the company's mindset so here's an example if you hear a present if you you know you see a presentation from a person um you know who's working at this company and they keep referring to their customers using derogatory terms like oh yeah we totally own these guys and they didn't know what was coming is that a person that you want operating

in your environment maybe it is maybe but maybe it isn't and and again it's not going to be i wouldn't say it's a make or break but it's another another way you can kind of gauge if this organization is a good fit for what you're trying to achieve um the other the other thing is uh like one time i um i was listening to a podcast episode and it was excellent but uh in the description of of what they ended up going through um i felt that they took it too far uh and that they might have um you know done too much evil so to speak and so like while it might have been a

great fit for that organization that they needed to have somebody who uh would really take it to the next level because that's what their threat model would suggest that their adversaries would do me personally i would say i wouldn't want to hire that person um again it's it's kind of um you know getting an idea as to what you would want the people to do uh would really help here uh next is you want to get an idea as to what kind of tools they use um just because companies are using off-the-shelf tools like metasploit or or something along those lines doesn't mean they're bad at it uh if you have the uh if you have the skills to

run it yourself you'd be doing it yourself so um but getting getting a list of the tools gives you an idea as to what the capabilities are and what they're actually going to be performing on your systems the other thing is when they're doing a vulnerability scan which almost all engagement should be doing uh it would be awesome if they use the vulnerability scanner that you do not use so it kind of gives you a free uh a verification of your own internal system um in in one engagement uh we found a pretty a pretty nasty uh false negative where our internal scanner didn't pick up this one vulnerability but the vendors did and it turned out that it was actually

there um next always you know price is a concern uh so what is their pricing model is it time is it based on the number of assets is it something else uh the other thing is if they're doing an on-site work if anyone ever does on-site work ever again uh you want to get an idea as to what the travel costs what might be so that way you don't get a huge surprise at the end of it the other thing is uh you know at the end of the day you're dealing with professionals so uh they probably do way more tests than you do so you know after going back and forth of the conversation

do they have any ideas that might help improve the scope uh do they you know they think you know there might be places to expand it or maybe tighten it up um or you know after after talks like okay yeah based on what you're talking about you know this web server or whatever it's probably not going to find anything but we do want to take a look at your active directory more you know so let's let's see if we can dedicate some more time to that that's all that's all useful um and then again if are there any other extras that uh that might be helpful so like then not everyone is a used car salesman

they're just trying to tack on useless things uh usually you call a company and say hey i want a pen test okay great but then when you know when you're talking you say oh you know what they have you know you have this is there anything else that we that might benefit uh like things like password audits or wireless um wireless assessments or you know just firewall rule reviews or something even if you just do it once to get a get a perspective and you don't do it all the time it's worth it to have a second set of eyes uh take a look okay so part two of the story the actual test happens

so if you remember in in the phoenix project in january sarah gets the boot and leaves parts unlimited so if they're testing a bunch of sarah systems she's not going to be able to be the scapegoat anymore so pentester's rs runs the test remotely then there was some delays on both sides the busy schedules you know end of the year kind of stuff uh they find um they find some default creds on on some store systems and they lead to uh exploiting compromise of one of the stores so a red team wins right you know great you've got findings and you know you can you can go and and try to remediate that now uh the

problem was that overall the experience wasn't really what they were expecting the communication was kind of problematic the delay there was some delays and then the report like they saw the the example report in the beginning of the phase and then when they got one it really didn't they didn't think it was it was good enough they you know they asked for screenshots and there really weren't any and it was just kind of like you know that's just what our tool provides so um you know that's that's what you've got so um pentester is our us starts calling back and say hey when are we gonna schedule the next one and parts unlimited says you know we're

not really we're not we don't we didn't really like the experience that that we had the last time we're gonna go with someone else and you know someone where's it's a good idea to rotate uh rotate vendors so we want to we want to try something else and therein lies a problem in their contract they signed a multi-year test deal even though but the the sales person said well yeah you can just get out of it whenever you want well yeah but the contract didn't actually say that and you guys didn't exercise the right to to switch when you're supposed to that's a bummer um you know so okay let's let's do let's do a second test thing uh let's

let's try it again this time let's not let's let's try to minimize our delays let's have somebody come on site so contracts they're important they suck but they're important at the end of the day the contract and the statement of work is what drives your engagement you want to make sure you get it right so one of the things that's important is there a non-disclosure agreement is it part of the sow or is it actually a separate thing and then what does it define as confidential and how does how is the confidentiality maintained these are things that are important to have in there because it sets expectations properly i've seen um i've seen uh proposals where

everything is confidential like you can't even say that we ever did business with you or other is where you know it's a lot more uh compartmentalized and says anything that only the things that we find out about your environment are uh are confidential and you know the conversations that we have things like that the the answer it kind of just depends on on on your environment next is the license to your deliverable who owns the content of the report and then who can see the completed reports uh again are the reports covered by the nda so you can't ever show anyone them are they only allowed to be used internally can you show them to

regulators can you just do anything you want with them uh there's uh you know i would say that there's a it's a good idea to be able to share it uh but if the if the license if the contract says you can't well that's something you might want to look into uh is there a section on dispute resolution that's important sometimes there isn't and if you have a dispute you're kind of stuck and it's just kind of like a he said she said situation uh having a well-defined dispute resolution process uh is is good and important

for multi-year deals double check how the severability works i would say if you if a you know if if a company full of all ns a former nsa agent says we're going to give you a 90 discount and uh but you have to use us for five years would you take it i don't know that you would because you've never worked with them and uh you don't know realistically how how they're gonna how good of a fit it's gonna be but even if you do you wanna double check the early termination clause sometimes it's you have to terminate within a certain amount of time after you get deliverables or you have to do it within

you know before any any anything any work is performed um it's you know it depends but it's a thing you should look at uh the other thing is you could just say don't do any multi-year deals uh i think some flexibility and you know you might get you might get some discounting involved if you do multi-year but again pay attention uh ensure the contract and the sow includes examples of attacks that are in scope and that aren't in scope so for example oftentimes you would exclude ddos attacks or anything specifically designed to take down your systems um that's you know usually the case but if you care about your availability and you want to test how

how well your systems behave under a ddos well you want to make sure that that's in scope or if you care about uh you know certain other classes of exploits or vulnerabilities uh you want to make sure that that they're in there at the very least you want to you want to have something that says what isn't in scope uh so that if you know if uh if someone accidentally or or mistakenly exceeds what they're supposed to causes a problem there's some protection for both sides and again realistically at the end of the day you want a contract to be fair um if it seems like it's weighted in one direction or another have a discussion maybe there's

something that they can they can change about it um other things uh as in most contract work uh weird things that you find in a contract might indicate that the company giving it to you has had a history of that problem and so maybe double check or you know don't just say oh yeah that's not a big deal uh ask like ask why that's in there or if they can give an explanation um if you know if you see it like a non-disparagement clause or or something where you know the damages include unlimited liability instead of uh some limit uh you could be exposing yourself to significant financial risk uh that you probably don't want to

uh realistically at the end of the day you're hiring a vendor and this is a lot of vendor due diligence um and you know some some work uh beforehand really will help out in the end oops so and then the other thing is you want to get examples of letters of authorizations otherwise known as get a jail free cards or the rules of engagement or something you want to get that in advance oftentimes you will have that kind of given to you the day of or the day before uh that the actual attacks would happen or just some some time before and they sometimes have terms and conditions that are in addition to what the original

sow was so you want to get a chance to take a look at those before you actually have to sign it you don't want you don't want the time you the day you sign it to be the first day that you see this again before before you engage double check some of that information and make sure that it's it it works for you okay so back to and testers r us excuse me so they gretchen you know they grudgingly go on with the second test so this time they're gonna they're gonna test uh a new a different system project unicorn pen testers rs comes on site and they they find there's a bunch of super critical vulnerabilities on all on

a lot of these systems we're talking you know cvss 10 scores all the systems were clones of each other for uh you know for the rapid scaling and they all contain this vulnerability that that somehow just wasn't noticed but there were no systems exploited or compromised so john and bill just think okay maybe it was a false positive so when they get the report uh it pretty much just says you know fix all these vulnerabilities but especially this critical vulnerability even though we couldn't exploit it um so john and bill push back and say well well if you couldn't exploit it why are you including it in the report uh if you know if it wasn't exploitable

it might not be a false positive you should you know that's that's probably not or the first thing that we should be focusing on and the only answer they can get is well that's you know that's that's the worst vulnerability you have you really need to fix that well oftentimes you know we'll get uh you're get into an arguing match with a vendor and you say well can you show us proof i'm sure that pen test companies hate it when the customer says show us the proof um but sometimes you really need to ask for it and in this case uh pentesters rs just said well we can really just give you you know the output of our tool that's

you know the raw output is as good as you're going to get you know that's that's what we've got so they put brent to work can fix everything brent can you guys can you take a look at the raw output of this pool at this tool well you know it's 10 000 pages of of weird like csv outputs that's formatted for printing that that's kind of annoying um but yeah i'll see what i can do okay so uh after a few days of working on it i figured out that they ran all these different exploits none of them were for software that we actually were running in project unicorn well what about that that cbss10

a vulnerability that they found the exploit for that's not on this list well what did they run they ran a bunch of attacks against apache we run engine x not apache in our systems yeah and we found in the raw vulnerability data that they knew that we don't run apache um so they ran the attacks anyway because so they know it wouldn't have worked so you know bill says well we wanted a pen test and they gave us a pencil just a bunch of number two um sometimes that's what happens reporting uh the report is the thing you care about at the end of the day uh because that's what's gonna that you're gonna take to

your board of directors to your auditors compliance whoever and it's also going to be the blueprint for how you fix whatever findings are so before the uh before they finish with all the work uh usually they're gonna tell you if something's super critical or if there's evidence of prior uh prior exploitation or or uh you know or anything else um so you want to get a good idea of what the findings are before the before they leave make sure that make sure that they find a vulnerability that exists in a lot of places that is exploitable and they think is super critical did they actually try to exploit it um because you know again like in the example if

they didn't and and it could be just a false positive

also make sure they try to document all the exploits that they did try or all the all the attacks or or whatever um so this is actually this is very useful uh for verifying that you actually have a working system um you know if something if they if they tried an attack and it failed and you can prove uh if you can provide evidence that you know some of your system blocked it or there was an alert that that triggered in a sock that's that means that your security program is working and you want to highlight that pen test reports shouldn't always be yep we own the system and we got domain admin in three clicks

and you know everyone's horrible at the job sometimes you you you know you want to say yep we're doing a good job and here's how and we made it really hard even though you know there were some minor findings or whatever um so you knew where all the bodies are buried and sometimes you find out that pen tests just find more bodies this is the great thing about having a third party um they think differently than internal folks they might find something that you had no idea was there or come up with a different a different way of exploiting a configuration or or whatever um you know again be prepared to fail uh no one's perfect so uh you know the

bad guys have to be right once the good guys have to be right every single time in order to prevent uh uh you know a security incident so um it's better that that that a pen test company does uh does the attacking and and shows you where the weaknesses are then a real attacker does it um you know just because you know someone got domain admin doesn't mean people should be losing their jobs uh so if that's a problem you might have just a company culture problem too um the other thing is if there are not a lot of findings make sure to include examples of the detections or the log files or alerts um where you show that the

attacks are are failing um because that helps um that helps a lot of avoiding worries that the company didn't actually do what they were supposed to uh you know it also gives the blue team kind of an atta boy uh that they're doing everything right um honestly one of the best engagements i've ever participated in uh was you know me and another pet and a pen tester kind of sitting in the same room and as they were carrying out the attack and they you know they said okay i'm doing x y and z and i said okay yep i'm seeing it log here and here and here and you know yep we're blocking you at this path and and

here and we took screenshots of that and included it in the report and it was honestly like the bat like it's literally the you know the idea of purple team and it was great because we showed that realistically that you know in this one system that we were testing there was you know it was set up exactly the way it could be the best way it could be uh and there was evidence to show that that was the case and there was evidence to show that it was repelling all these sophisticated attacks which really gave everybody the warm and fuzzy feeling that we could bring it live and it was going to work the other thing is that

the remediation plan that you get at the end of the report realistically needs to be doable um so a lot of time like often times uh the companies i've dealt with they are very amenable to uh you know to say something like you know well what do you suggest we do what we recommend you know recommend that you do for this kind of thing if it's you know if it's kind of off the wall so you know if you scope to test a physical test and um you know someone uses a missile launcher to break through a wall and get into your data center you know probably a pentas company is going to say you really need to armor

that wall but realistically that's not going to happen if that's not real if that if no one thinks that's a realist you know uh a worthwhile effort so instead you might say okay well great yeah you got the missile launcher uh and blew up the wall how about you say that we have an awareness campaign so that if people see a missile launcher they say something that might be you know that's kind of uh a way around it uh if you know again the company is probably gonna tell you you know the best possible way but realistically sometimes you kind of need to meet in the middle as far as uh something that's actually achievable

uh also if you scope the pen test where a missile launcher is in scope kudos you know but that's that's actually all i have for now and i i feel like it kind of took an abrupt end and i apologize but we're kind of running close on time i want to give a special shout out and thanks to investigator check uh kathy she helped me put together my abstracts for when i was uh submitting this uh really helped me solidify it and uh p9 patrick always uh awesome uh person for bouncing questions and ideas off of and obviously he organizes this and other conferences awesome guy in general overall so finally uh are there any questions

hey dmitry hey okay yeah that talk was fantastic we only have a couple minutes for questions um before we get into the next no it's totally fine i personally just want to say this is like shameless plug but i used to work for a pen testing firm so i was super appreciative of how much depth you went into on the scoping side of things because i feel like we often have to evangelize security to some of these companies to get them to actually sign off on some of the scope that we want to do as pen testers and so i just really really appreciated it um and with that there was one question that came from

frobius which is how do we best educate people organizations or companies that in this case it's running a nessa scan and printing the report is not a pen test um but in general like how do we evangelize security and help get them to actually like sign up for the work that we want to do so honestly in my experience and and so this is just me i'm guarantee you someone is going to tell you that this is absolutely wrong and that's perfectly fine if this was an in-person thing i'd say let's give me a ginger ale and we'll talk about it later sometimes you really just need to be scared into it um and not necessarily scared but if you

can you know if you can prove that that the net that the need exists by you know running one of the tools yourself uh or you know or at least you know explain to someone in concise terms that you know okay here is exactly how i would do it and i would take this one vulnerability over here and then use that to get a you know a hash and then pass the hash over there that's not something that you realistically can detect with a vulnerability scanner vulnerability scanner just says this one problem exists on this one computer or all these computers it really takes you know a human person um to to put it all together and say here's

an attack path that you know that will realistically you know cause you problems that were just single vulnerabilities on their own might not necessarily rise to that level yeah i totally agree again like nobody asked for my opinion i'm just gonna give it to everybody but um like i have seen so many examples especially on network contests where you know a nessa scan might print out a like a bunch of like low and medium vulnerabilities and then we can take those low medium vulnerabilities and daisy chain them together to something that's critical so you wouldn't have that human element telling you that all of these are kind of intertwined and can result in some privilege escalation and really bad

vulnerabilities if you were just looking at this printout right yeah i mean breach and attack simulation is a new kind of world where theoretically they can do that but i think realistically there's always going to need to be a person there i agree i agree but i'm biased um we have time for one more question which is from yo diggity uh they are curious about your thoughts on crowdsource pen testing options that are starting to gain momentum um for example bug crowd has several varieties hacker one uh bug bounty programs what do you think so i think it's um i think it's it's a great direction to go and i they're realistically um they're a good fit for a lot of

organizations and not some others i feel like you need to have a significant level of maturity to really get the most use out of it and may and some critical massive size so like my experience has been a lot with midsize and small organizations so most of the companies that i've dealt with probably don't rise to the level of needing a bug bounty program uh but it's um i think it's worthwhile and you know the problem is you end up with with uh people who are not being um you don't want to end up as being someone who's not playing by the rules or not being like a good corporate citizen where you're not really giving the the

you know the bug bounty people uh you know they're they're due so to speak um i'm not a fan of companies that just kind of hide their you know stick their heads in the sand and avoid problems and um it's been publicized where like some you know that some i guess the if we could fix the abuse of the bug bounty system i think that would be overall much better um but it's it's worthwhile again it's it's you all these kinds of questions that i brought up in here and you should be asking of of that provider i asked them how they you know how they would how they would tackle it and how it

works and how how it really melts with what your processes are totally i couldn't agree more dimitri i really appreciate it you absolutely blew us away with that talk i was not even surprised to to learn that because i had very high expectations and you exceeded them um but really appreciate your time today thank you so much and please pop into the discord channel for any additional questions

and this is bibliography thank you so much thank you so much everyone