Automation of Penetration Testing and the Future

welcome to proving grounds I'd like to start out by thanking our sponsors on the stellar Staller level ver spike fertility tenable Amazon and source of knowledge please visit them out in the chill-out lounge so I want to remind everyone this track is being recorded so if you have a questions at the end please use the mic at the front or I'll run a mic to you at the at the end of the talk so our next talk is on the automation of penetration testing and the future by Hayden Johnson let's all welcome Hayden

hi everyone thanks for coming a bit of a full-house bit more difficult than practicing with three or four people so I'm feeling really fortunate to be here mind blown actually excited I title the talk is automation of pen testing what way in the future 3 second Who am I I do pen testing I have the OSTP certification in GXP n recently my approach is to work with blue teams instead of just throwing the report over I tried actually helped security and sort of break things and be awesome so I guess let's start three years ago I was back home in Melbourne with someone telling me to do audit because security was not where the money is and they were

pushing me to do audit and cybersecurity was career suicide but here I am now presenting in InfoSec and you guys are all lovely so a big finger to those guys anyway we're not here to hear my job laughter starting a career in InfoSec we're here to learn about my jaded thoughts on the way of automation of pen testing so thank you in advance for listening and I hope I can caught some knowledge that's valuable so last year I was at Derby con 5 a panel discussion raised concerns about the future of pen testing the pen testers V A's and the integrity of the practice this quickly covered things like dentists puppy mills what's a VA what's a pen test both the

same thing and how they overlap each other work tomorrow so so this is just a my thoughts on this so just a two-second note this is on my own time and I'm so all my thoughts on my own they don't represent unicorns my company or anything else so what am i planning to quickly cover covering the trend of automation pen testing pen test puppy mills small beat business reasons for automation pen testers exploit devs and what they hold for the future and what we can try to do a 5 back so I guess I'll start this culture of good enough we all know it right we want a VA or a pen test just to tick the box we want

the report so we have the report right but like you and I will we actually see anyone do anything to fix it or any action on the recommendations this is extremely frustrating for myself and probably all you guys all of this is pushing the commoditization of the am pen testing has been for years anyone can do VA in pen tests right run a tool easy mode I guess at least that's what everyone thinks this is the trend of automation so those selling pen testing want to do the smallest amount for the most money and those getting pen tested want the minimalist completed to make them not liable for any breaches they want to be like we were pwned but we

were compliant so target for each years ago they were PCI compliant if I can remember right they want their automated scan and this is pushing the dollars and profit and the skills needed that way less funding for more skills and equipment and infrastructure pen tests have become V ace so I'm unsure what's actually being sold as a VA maybe just a scan and this is now penetration testing and how it's become commoditized we have packages a B and C we only sell these instead of customizing each engagement to the client we just have the packages to make it easy there is no confidence in penetration testing and customers feel that they will be given a glorified

scam with half-hearted posts on the network I think that's a quote from a red teamer in Montreal this is where pentest puppy mills come in we have the automation pushing down the price so now we have plenty of hand test puppy mills fighting for the cheapest price when companies want better security they choose price not security yes you heard that right instead of choosing skill and talent the decision is made on price this is ludicrous and quite a big problem it's a big problem because like myself there's less money for an engagement you only get minimal work smaller engagements smaller man-hours mean scanning yes less time pushes you to make your methodology more efficient so

you're not just wasting time in a crappy pencil style but that one is easily crossed when it's drawing down into one or two days and you have multiple subnets it does not matter if you're elite with all the social skills minimal budget means you're just going to scan it's also another big problem because different skills mean different findings different exploits different vulnerabilities these don't really happen in automated findings they're all known so the cheapest place is not going to have any great talent at all and if they somehow do they're not going to keep it for long so as the slide shows we just want to scan and scan make the report look nice up look nicer again and

then send it I've experienced this as anyone else want to put their hand up to say they've just done a scan as if here a pen test yes we've all done it and it's extremely frustrating for us passionate people so this also then leads to outsourcing some countries have cheaper labor labor so makes sense to use them right back when I was starting well before I started same company was the outsourcing work to like Argentina or India as like some extended delivery service outsourcing pen testing doesn't really make sense because based on our skill so they were doing it for a country known for extremely cheap rates interesting idea so the cheaper guys would do the pen test or the VA attempt

to write up at the report the technical pen testers back at base would then review the reports and fix them up so the technical people of base were reviewing reports the passionate and technical people weren't doing the work they were just reviewing the reports that would make me hate my life so it's a manager's job so after a while I guess you can tell where I'm going with this those people left the company one by one well probably because we're all close-knit it was like in big groups so then those people were doing the reports so no local talent or in-person talent means no market share the talent pool moved on so this was a game for the

short-term for the company that really a big loss in the long term so you just can't outsource pentesting unless you do it appropriately and another story on this I was once told to go to all client to set up some car we jump boxes so I'm like a new grad couple weeks and excited go set it up talk to the coin yeah I'm gonna be doing the pen testing I get back to base in the man just like yeah that's that's free India Argentina what so they're quiet thought I was doing the work but no like ah so pentest popping mills and outsourcing really results in unmotivated skilled and unmotivated so we have this automation of pen testing

trend pen test popping rules as a result so what are the basic reasons or some of the reasons the big businesses or small businesses ask for automation businesses are either the symptom or the cause I haven't really decided or added blame but it's a problem it's a bit hard for small companies they do not have free money some may just be surviving month to month and any profit is really needed they do not have money to throw problems they do not have a budget for a security team they just want to survive day to day mom-and-pop shop making some living security in general is probably not in their mindset the budget allows a sysadmin that does all the things they

just want to survive and if they have any talent they're not going to keep it for long I'm gonna use my nerdiness and like it to MMO gills if anyone plays war two Warcraft used to play Warcraft you start as a noob you join in you join a noob guild you get as good gear as you can at that guild then you got to a medium guild and you get better and better gear to you max out then you jump to another guild and buy guild I mean company we keep jumping there more money or opportunities and small companies just just can't compete with this so big companies well big companies are not in a really bad situation at all they're

just big AKA slow and dumb and even dumbest shareholders they throw money at all the things they have deep complex politics to dance around for anything you want to do and a result anything for them move really slow they're required justification for anything complex networks like some CBC attacks legacy everything as a result and did I mention air politics politics all the things being big and all they have old policies so you Americans that have your declaration of independence it was written a little long ago in mind for a so what it meant back then probably doesn't really mean or take into account what's here now so I'm sure was written with the idea of free speech but what

about your Wi-Fi connecting to a Darth Vader to spur probably not it so on summary small and big businesses just small businesses just want to keep the network up big businesses just want the problem to go away so they can keep making money unfortunately this issue is the same with blue teamers companies will want to buy the blue the blinky boxes because hey the blinky box does all the things right from the vendors but it's also cheaper than hiring staff on save fifty thousand dollars a year for four years but the blinky box is not going to catch anything in four years or even if it does the day you install it so we have

automation of pen testing trend pen test puppy mills and business reasons for this so what does this result in a big bag of confusion on what a pen test and vulnerability assessment is in the eyes of the consumer there is little difference between a vulnerability assessment and a penetration test they do notice pricing so back in the day I tweeted that we needed some clarity on terms because everything is everything and I was just flabbergasted at all the different terms there are and what means what so Twitter friend created a blog to help clarify the definitions as you can see on this page there are so many terms and obviously not just related to pen

testing but you get the idea if we cannot understand the key difference between a VA and a pen test how on earth can we then understand adversarial emulation or even the simple purple team exercise we just don't so I'm sure everyone's like oh us person a what a pen test is you get one answer you ask for the person B what a pen test is another answer there's no definitions now it's our industry profit is a big motivator for an unscrupulous agenda and InfoSec is no stranger when it comes to shady practices so v fear uncertainty and doubt and snake oil our industry will take an ambiguous term like penetration testing and reduce it down

to the lowest denominator to make the most profit pen advanced pen penetration testing is different from pen testing right well you would hope all pen tests are advanced that's the point of their job the other part of problem is the word penetration testing some have used goal-oriented penetration testing to differentiate this from a simple vulnerability scan or assessment depending on the goals just so that penetration tests differ and then red teaming is a whole kettle of fish on its own some companies are selling red teaming to claim back the old penetration testing which is in effect making more dollars of what we used to do with a pen test so we it's all over the place so we have this trend of

automation scanning and a confusion on terms scanners run on finding vulnerabilities so what will happen to the people that find the vulnerabilities that go into the scanners aka what will happen to the exploit developers we have to be able to run scanning tools as a pen tester company environments are too big to do without so we do need automation we don't just but we don't just scan and then say here's the NASA scholar's report go go go do it we understand the findings for each business risk qualitative and quantitative risks so a credit card company is going to care about different things then there's a hospital with health data will point in case a recent

rape ransomware attacks with hospitals just different different things that's pretty simple so we analyze the dollar amount of the impact the qualitative risk the chance of the occurrence with the goal of helping the business understand their business risk if you just run a scan it's not much value instead of if we're just saying hacker can get shelled um pwned basically we also need some scripting skills understand your programming and networking 12 allow us to edit the tools some basic exploit development is used here so what happens through exploit developers so with exploit development its bug hunting finding vulnerabilities and all the things these are top tier skills and these are needed to keep feeding the lower skills the economy of

ear and pen testing with not only new things to do but to learn because we're all interested and exploit developers cycle is that they find an exploit or develop one better than a pop that just crashes here application its then put into the scanners the scalability because that exploit developer can't go to volvo exploits in every environment so then it feeds into the vulnerability scanner and while we're scanning all the things the exploit developer tries to find other vulnerabilities and it keeps this healthy life cycle going we need this skill we need to keep feeding the scanners and Infotech is always changing technology changes so we need them in a nutshell we have talent from easy to hard or wave this spectrum

there is motivation to move up the path and keep learning but with this push for automation what do you see happening here the talent currently all over the spectrum automation can build exploits right automation can fix open source tools if they don't work well the talent pool is going to sink gonna shrink at least in the white hat world since there will be the jobs in scanning there'll be little demand for exploit developers no one will be able to afford them and no one is going to want to learn those skills because it takes quite a lot of time there's no reward and passion can only get you so far before you say hey I

really need a job to feed myself and my family get bread on the table less there'll be less exploit developers and bug hunters less vulnerability researchers meaning less security for everyone or exploit developers will find things that whether they sell them to the vendors or the white hat world probably not so we're going to lose all the talent so in short scanning all the things there's a confusion on terms and less money to go around the world is fubar there'll be less money to pay the talent so they'll be cheaper faster less skilled pen tests being executed meaningless vulnerabilities found if any less vulnerabilities being identified means more unknown vulnerabilities but so criminals will find all the things

and sell them on the black market there will be lots of vulnerabilities just a black hat market will be booming with them so since we have the penetrate automation of pen testing confusion on terms and pen tests puppy mills and the world going to chip what can I do or what can you do to help change this as a pen tester or a red team oh we need to educate managers we need to educate clients and we need to promote the valuable security of the pen test red team over just scanning all the things

so we need clear understanding of terms a vulnerability assessment is used to identify and categorize the vulnerabilities so for an organization that lets them quickly see the overall posture their systems on their network it might be the very first time they have a network diagram or a list of the items but the key thing with the VA is that it's a listing of prioritized vulnerabilities a penetration test is focused on breaching their security controls to achieve a goal so such as Active Directory or PII data so Jerrod freights was on breaking down security recently very recently and his thing was a penetration test is to grab some sort of control show the real impact of a

penetration test so in a side-by-side comparison you guys can see it a list which is great if you're a really low material type of organization and then a penetration test is goal-oriented so if vulnerability assessment uses necess and bird I had someone say a burp suite scan is a penetration test and that's the term again so with penetration testing we do use this as we do use burp use Metasploit reverse shells sequel map it's a whole different kettle of fish so we need to explain to our managers that yes VA is easy and cheap right has it has its own value but a pen test can add much more and probably bring in more money and then

you can show the real impact actually have noted that just before as far as looking at the slide but it shows the real impact it can help people get budget for their security teams it can say hey I've told you about this you didn't think it was anything but they just popped the shell or were able to spill data just by chaining a few exploits so you can understand the real impact of bypassing security controls so all-in-all I ask everyone to help fight the good fight and if you have any questions hit me up on Twitter or throw them out here thank you very much

does he we have any questions

with b-sides in Def Con and blackhat getting so popular do you do you think that we are not gaining more talent or do you think that the security is we're more aware of security problems and that is more of the problem rather than getting talent we just can't keep up with it oh good question probably a mix there's obviously more awareness of security button every day you saw a girl I was dating was like on you do block picking at a conference like the average user has absolutely no idea so from a pen testing point of view or even say client the way talking to doesn't have that idea that they want a VA they want

their security check this education but then you're also limited to what they want versus what they should want and everything like that so you mentioned keeping track and hiring good talent but in the day where vendors will sell you certifications and paper mills just pump out cyber security master's degrees how do you suggest we identify and maintain that talent has the company how to maintain the talent or how to go through HR and identify crappy suits second one get your OS VP done no but like even the GX PN was a multiple-choice exam and because I did that always sleepy really prepared me for that I think something more practical and hands-on is a lot

better than something multiple-choice regarding crappy certificates I'm not sure if we need a governing body on your pentester needs this or there's been a lot of chat about interviews and how you should do it with tests and hands on I think there was a couple online gave the way the type of questions they asked and practical hands-on exercises for their job interviews so maybe along that line and you can test skills versus your what's a TCP packet doing this pin a cat things like thank you yeah so I really like that you differentiated penetration testing as a goal based I like to explain it that a pen test provides a path yes as opposed to a listing of issues and I think

that's sort of what he Illustrated there so you said to discuss I think one thing that you didn't really cover is something that's in the middle of those two which we'd like to commonly referred to as a security assessment may incorporate the vulnerability assessment with validation with validation yes but additionally looking for configuration and design flaws so something that won't assessment doesn't necessarily test is something like egress filtering yeah which is a network based security control but isn't necessarily a vulnerability you know isn't there's not gonna be a message in for egress yeah so for even processing procedures and things like that and I think some of those things like a security assessment you can develop a

methodology that can be done more quickly and for those smaller companies I don't have the money to pay for multiple people and some sort of threat emulation exercise like a pen test can be able to afford to get a lot more value out of than just a list of vulnerabilities that doesn't necessarily really improve their overall security posture yeah and that's the frustration I certainly agree that I shouldn't have to dump your domain control passwords to tell you you should make the complexity longer any of puzzlers thank you all right I think we have time for one more question anyone not really question but I will like to fill in normally on your comments and

that's the threat assessment how do you put the vulnerabilities that you find in context so they can be used in the penetration off so that you can read your assets it's also in in between I mean if you think of a risk assessment then we have like your your assets and you define your assets and how much you're worth the technical but if you if you don't really care about how much stuff costs and if you keep it on the technical level then I usually call it a threat assessment what vulnerabilities will always be there credentials will always be found so if you have that and all the other vulnerability that you found finding a

vulnerability assessment how does that lead to a compromising an asset it's really good question I'm gonna take a mulligan on that no but I think it's like risk appetite and everything like that based on your vulnerability is what you accept what don't you accept different mitigations for prevention I know with some PCI or audit it's like yes this is a risk that we have dis mitigating control and we can detect prevent I'm not too sure how to answer your question but that's my being around the bush answer I guess thank you