Whatever Happened Last Time, It Wasn’t A Penetration Test by Joe Sarkisian

all right awesome thanks for coming guys um came here from boston massachusetts my company paid three grand from here coming around trip to give the stock so what what to them um my name is joe starkision uh i'm from den secure which is our brand new newly branded like offshoot we have like we're a financial company wolfen company we do like auditing and stuff like that so i like to say we're a security company that's essentially bolted to the side of an audit firm so it's cool that we got our new brand done secure um i've been there for about four years i'm the lead penetration tester there got a team of about five people um again this is our i don't think anybody from work watching this is a boring slide it's just about my parent company wolfen company a lot of people that work there if you need audit whatever come to us ideally come for a pen test uh about us again social engineering advanced security assessments so just real quick to touch on that we don't say anything's a red team because most people don't need a red team they just think they do because it sounds cool so we called it advanced security assessment it's objective and goal based right so that's that we do threat emulation but let's talk about why we're here how many people in this room are pen testers you got few hands okay how many have had to buy a pen test on behalf of their company okay a few hands awesome all right so we got like an even mix there okay so it's gonna speak to some of you hopefully so i've been there for four years in this entire time we've had this problem of we have a new client they're not sure what a pen test entails they've had one in the back like i don't know a year or two right like maybe they're regulating they have to have it every year i don't know whatever it may be and then they contract with us and they're like okay what's a pen test essentially and it's like didn't you have somebody do this for you like a year ago like that they did not like put you through the paces and tell you how this works whatever right and then so we're it's almost like we're starting from scratch and we have to like hand hold people through this process and it's like we're fine doing that but we need to get better as a as an industry as a as a security industry where that starts to be understood when we come in the door right it's very important so it's still poorly understood right by practitioners by clients uh by people who aren't even in the industry obviously it's just poorly understood i tell people what i do and they like walk away because they think i'm a criminal right like that's not what we do um as a serious consequence if you get it wrong right like false sense of security for one um there are a lot of vendors in the u.s i don't know about uk eu i see not like people like yeah people suck at that okay so there's people doing this that don't actually perform them like the classic von scan sold as a penetration test right those things are not synonymous um and we're here because it annoys me and i want to tell you about it uh this tends to lead a lot of this tends to lead to the following like responses that we get either during a scope call or an exit meeting after we've like given them a report to go through and it's every single time but we let you in so we're secure right because you had to let you in i'll talk more about that or that's not a realistic scenario like okay uh our mssp would have stopped you this is my favorite one why didn't they um does everybody know what mssp is i don't know if that's like translates are cool um this report does not actually adequately reflect our environment okay but we're tracking that issue okay risk acceptance awesome our report was clean last year okay why didn't the previous vendor find this this has no answer i can't answer this question but i get asked it a lot so here's a scenario company aids regulators require pena a pentest annually at fdic in america you have to have pentest every year right uh the bigwigs don't get security at company a so there's very little budget to have this done uh they look for a cheap vendor right there's plenty of those there are surprisingly tons to choose from at their price point if you google penetration testing services when i do it at home the top google result will be like 99 penetration test and i'm like okay and then like a lot of the rest of the first page is something similar like that i'm willing to bet you're not getting a pen chest for 99 okay uh they pick the one close to the cheapest fair i mean look at their situation right who wants to guess how this turns out dumpster fire okay so they finished the test and here the report looks like this it's either one of two things there's no or few findings right or a massive spreadsheet of vulnerabilities that was not tested for or for validation right if you get handed one of these as someone who's buying a pen test you didn't get a pen test you get a vulnerability scan and they're telling you it was a penetration test right and we're gonna go over like things to look for if you're trying to get a pen if you need to buy a pen test but i'll just say it right now like the first question you should ask if somebody sold you a penetration test and hands you one of these is what did you exploit from this list show me and they won't be able to now they have either a false sense of security because they have very few if any findings right everything's ship shaped good to go uh or new job openings after you slam the it department with a massive report that they now have to sift through and they're not going to get through either that audit year or probably not the one after that okay so it builds up if you're using the wrong vendor over time so why did this happen lack of buy-in from the top right the sea level is like oh god security is a pain in the ass like this is too expensive whatever it may be like we see this as a cost center uh lack of budget because the c levels don't understand why this needs a budget which leads to a crappy vendor as we talked about which leads to poor work false positives false negatives which are even worse leading to a false sense of security massive i already say that or massive yeah okay my massive pile of monetary work leading to a breach right because you think you're secure you don't have budget nobody gets security at your organization and this is what happens uh-oh sorry surely this will happen again next time right next step so another scenario company b's regulators also require an annual pen test the big ways understand security there so there is solid budget they do research first and they send rfps to people that they have researched that have seemingly good track records this takes time right but the rate but company b has people that understand that this is important and therefore the people who are going to make these decisions and pay for these services have the time necessary to do it right there are surprisingly very few of these to choose from in america there's a company called secure ideas anybody know who that is security is kevin johnson owns it okay on his and i i love their company for this i love him for this on their own web page they sell pentest right they do a lot of web app um api stuff like that but they'd also do like social engineering assume breach whatever it may be and honestly it literally says if you don't go with us here are the seven other people you should buy a penetration test from and he names like but like trusted tech black hills like all the yeah i see everybody knows those companies all the all the good ones right i wish he would put us as the eighth but [Music] um and they picked one that makes sense for them cool once he gets how this turns out and i forgot to put a cool meme here after this so you're just gonna have to guess that it went well so this penetration test finishes and the report looks like this validated useful information right so it wasn't just here's all your scan results and uh we'll see you later it was here are the things that matter so like how many people in here are responsible for like remediating findings okay two okay well all right i'm going to pick on you guys i'm going to pick on you guys then has this happen to you where they just slap a spreadsheet in your face okay good does it happen to you in previous life at another company yeah yeah yeah so a lot of those scans if anybody's ever run a vulnerability scan you'll come back with like stuff that's like critical high whatever critical would be like oh my god you get ms-17010 right and that's critical i'll be honest but some of that stuff in there is like you might see like spectre and meltdown type things in there that like really like it's critical based on like cvss or whatever the metric is that they're using but if i look at that report as a penetration like i don't give a crap about that right i care about the informational that says that like smb signing is not enforced okay or smbv1 is in use or lmnr right stuff like that now if somebody is handsy with scan you're not going to like if your sop or an sla rather in an organization is we fix criticals within x amount of time we fix highs whatever medium whatever they're never getting informational which is where the gold is as a pen tester or as a bad guy right so just think about that if you ever end up in the situation where somebody hands you that dock um anyway i'll get back to this so uh it proves risk right so we're intensely proving risk we're showing screenshots we're saying i did that like i don't believe you did that here's a screenshot showing i did that right i did that that happened in your organization that was part of the penetration test here's why it matters uh it helps the company make good budgeting decisions in the future right you can say look at in the report like okay sure you have a knack and you have a firewall and you have this like those things work well here's the thing that doesn't focus on that if you need budget we can help you get budget because we told you you have a problem there right you have a hole in your in your security so we can help with that um it informs stakeholders of day-to-day staff about weird security issues they won't notice so that's kind of what i was getting back to like the vol scan with like the informationals where like i'm gonna pay the most attention as a pen tester or a bad guy um you can kind of re-re-risk those things to people and say look i know that you have to patch and that's part of your sla but you should probably pay attention to these other things right fence and depth um and there is no option b right because it went well like things things went well there's no other problem here right now company b has a solid point of reference um their team is bought into the security process right like if you have a good vendor they're not just gonna like even if they do a good pen test they're not just like slam a report on your desk and walk out the door and never talk to you until next year right what they should be doing if you're buying this type of work um is in a way hand-holding you through remediation and i don't mean they're like also being paid to come back and fix everything that you found because then there's a conflict of interest there potentially um but what they are doing is saying like here's the recommendation right like here is what you can do like not quick fix but like the general idea of what you need to fix and if you have more questions like we'll provide you more documentation um we'll we'll tell you like yeah i know you want to fix it this way but there's a reason that that's not going to truly remediate the problem like you need to do actually x y and z as well um so there's you know you should be able to pick up the phone two weeks later and say hey i have a question i know we already had the exit meeting but can you answer this yeah i can right like i'm we're not going to bill you because you have a question for 10 minutes of my time any company not just us um and there's less likelihood of a nasty reach like yeah you're going to probably sooner or later if you're somebody of interest you're going to have a security issue you're going to potentially get breached whatever like there's no magic bullet there but there's just less likelihood of it being bad right so i am blowing through this talk it's also the first time i've ever done this talk so that's why i'm blowing through it uh we've kind of talked about this a little bit um do not shop for the lowest cost provider okay like again i part of the reason i'm giving this talk is because as part of our pen test we're sometimes given a prior year report for other like whatever vendor they went with last time that was cheaper and oftentimes it'll like it'll be just like 20 pages of just a wall of like text and it's like about our company about the people here about our culture and it's like there's no executive summary where like somebody who's gonna make decisions just look at it and be like in two minutes tell me what's going on here and then i can leave and go back to my next meeting right it's just fluff and then you get past the fluff and then there's no narrative right so there's no section that says we did this and then we did this leading to this and that's a finding right and then we did that and blah blah blah and that likes we kind of spidered out here and we got access to this data and whatever it may have been right there's none of that there might be sorry one more time how many people here have ever received a penetration test report five six people okay how often do you get a finding that says like uh tls version like one is not is not enforced or enabled and they have that as like a high risk vulnerability has anybody ever seen that you guys have seen that yeah randall to me that says they couldn't find anything else and they're like make it a high because we got nothing else for you um that's kind of what you get with the lowest college providers right that's like we'll scan we'll do the and then like we'll poke it like whatever we find right like whatever the scanner tells us like we'll poke at if we can't do anything with it oh well look into the vendors you're considering right so like take the time to research um again when you google for providers you're gonna find like the first page is gonna be taken up by people who paid for like the most google ads or like however that works and they have like all of their like super low pricing and all of their like elite hacker speak can you go to their website and it's like bits and bytes all over the place and you know all of the typical jargon and lingo look into it i would say i mean is anybody else here from the us nice one person okay i don't know i mean i know that sygenta obviously they do their stuff around the globe a lot of black hills trusted sect do as well but they're also very expensive right um ask for references right so like if you do the rfp process and you're like i need a pen test uh and you pick say three places those places should be able to provide you references right so like who have you spoken to you did a good job for who's willing to allow them to be who's willing to be reached out to in order to like talk about how that process went for them and kind of what they got out of it you know complaints or the good stuff the bad stuff whatever um if they don't want to give your references that should be a red flag unless of course all they do is work for like the dod or something like that and like i could tell you but then i'd have to kill you pay attention to communication this is big so let's say you've you've done the rfp thing right and then you're gonna have an rfp call right so you're gonna talk to these people for the first time like on the phone you have your stakeholders there right if they don't have questions for you about the environment you know like what are we looking at here like you tell them i need an internal penetration test and they say okay like that's not the end of that conversation right there's rules of engagement that have to take place there's like okay what does that environment look like i there have been times where i have not been on one of those calls as a lead pen tester not my current job and the the i'm not on that call the work gets sold i'm now doing the work and i say i call the client or we're now we're going to have like our pre like engagement scope call just to like make sure we're on the same page like what are we actually doing here and i realized that nobody from the manager the partner whatever took any notes when they had the rfp call and then sold the work and i'm like i don't know what i'm doing for this client because we didn't have that chain right so that needs to happen like there needs to be that chain of communication so things questions be like okay what do we need like how flat is your network right what do you have in the environment uh there are times where i've gone in thinking i'm doing an internal penis like we don't have any servers like everything's in the cloud and i'm like [ __ ] okay i changed things uh did they tell you about their process right so like if they don't say okay well based on what you're telling us and what your network looks like here's like what we think makes the most sense for you right if you say we have this we have active directory right we have whatever x amount of servers we have workstations this is out of scope this is in scope like whatever right okay but what about social engineering and it's like we we asked for an internal pen test like why are we trying to sell social engineering now right i mean it's like if we know what you want why are we trying to then like build you out of more money for something you don't want or maybe don't even need um it's part of the sales process sometimes but like if you're like way over here when you're trying to like scope these things and sell people things like it feels kind of dicey i guess you could say but let's say it's the internal pen test what do they do for that internal pen test like do they just like throw mask in at your entire network and just hold at like 10 gigabytes a second or something like that like rate and just hope for the best right you need to know what they're going to do what kind of tools are they going to use are they going to uh you know exploit things on business critical systems without telling you first right like if you fi if we find an exploit we think it's valuable or vulnerable we need to like tell the client before we do it because it could be a problem uh do you know who's performing the work so who here knows like what i mean this is like an american tax thing like anybody know what a 1099 is contract so basically it's i found out recently talking to other people who've been in the industry far longer than i am than i have that uh groups like ours right like so there's a lot of cpa firms right and all of them are like we do audit we do tech whatever and they're all trying to get into security because it's like really like you make a lot of money doing it right if you're the firm so what they'll do is they'll put on their website we do penetration testing right call us if you need a penetration test whatever how many people do you think work for that cpa firm that actually is a penetration tester right i found out that a lot of them are 1099s which means they are subcontracted out from like god knows where to perform this work they're not vetted probably right they're just like oh i do bug bounties and i guess like i'll send my resume to like all of these places and hope somebody bites and once and will let me do work for them on a contract basis so if you don't if you ask like okay who's performing the work are they contractors like how does this work do you have an in-house team whatever it may be i