Night Of The Living Dead Pentest by Rory McCune

talk today is Nate of the Living Dead pen test hopefully reasons why I've called that that will become clear as we go along I failed 20 questions feel free to ask as we go along or keep up the hand or just silence I the way it works it's late afternoon so very important note just to start off with as I work in the Penta industry these views are my own personal views this is stuff I kind of picked up across a variety of jobs a number of different companies at a number of different industries so it's all my own personal views not necessarily those of any employer past present or future so why is this called

it and test zombies so about six and a half years ago now besides London with the very first pea-sized London in 2011 they were kind of struggling for talks back then it was just a star that besides movement in the UK so I foolishly put in the talk I'm thinking there's no way this will get accepted and then they accepted it and it was just something that was annoying me either talk called pentesting must die and unfortunately it appears it's not dead yet I don't know what how that worked out it's obviously my top field so it's not dead yet it must be a zombie and it's wandering around still doing zombie pen testing so here I am I would

try and kill it off once and for all um to talk about why I'm gonna try and kill off once at all first off I'm gonna have to talk about what what is pen testing right that sounds like a really obvious and easy thing but I'm hopefully going to convince you that that's the problem that we actually don't know what pen testing is so then I'm going to talk about why is the fact that we don't know what it is a problem and then because there's not a lot of point in just saying here everything's terrible this is all broken without trying to fix it I'm gonna try and convince you here's a different way of doing this

that's better so my pen test journey usually when I do talks I kind of have a very brief about me slate because it's not really very relevant and we flash by it and I thought about this picture the Halen's where I live which looks very pretty but for this talk it's actually kind of relevant because I'm talking about my experience across a number of companies and jobs I started out in security 17 years ago I know none of you believed it by looking at me but yeah I've been in the industry for about 17 years now I started working for an chat services company I was an IT security and it was for our own internet banks so

we obviously we have pen tests done so for then I was I was a fire of pen testing services and we'd take the results after the pen test company handed them to us and we would then talk to our developers and say how are we going to address these problems and fix them after that I moved to one of what's called the big four which are big consultancy companies and I worked as a consultant there so here I was selling consultancy services which kind of included pen testing then I worked for a couple of banks in the UK I used to go to Pudsey ever known as the punchy data centers I was there regularly

and I ran their pen test service line so essentially here I was doing pen tests myself for the bank and I was also dealing with the reports that came in and so sailing getting external companies coming in to work and then handling the reports and they came back so I kind of get to solve the middle point of I have to explain what these pens after people have done to hold the developers and project managers and business people then I worked for a big provider of pen testing services and then I went and actually ran a couple of small consulting companies or helped on them we saw pen testing services so I got to try and sell this stuff for the first

time you know I should have ago to come in here say here you want to buy my pen testing services and these days I'm back working from a large pen testing company again so I'm back doing delivery work now the point of that kind of like long list of jobs is the three perspectives really I want to talk about there's a perspective of a buyer all right so I'm buying pen testing services I wanted to get someone introduces work for me and deliver that there's a deliverer you know I've been told right next to be doing on-site healing a pen test get yourself done and there's a seller which is someone's put a quote ask you to come

in for a meeting and they want you to sell them something which is pen testing and all three of these can I give you a bit of a different picture when you try and do them about what the problems are so what is it well that's the first question we have to answer any further Wikipedia in 2011 so I went back to look to my own slide deck and I said back there I went looked at Wikipedia it's the obvious thing to do when you're trying to define a term and this is what it said I kind of highlighted who I think is the important phrase but basically this is a meant penetration test occasionally pen test is a method

of evaluating security of computer system or network by simulating attack from a malicious source known as a black hat hacker cracker thankfully the term cracker has dropped out of the security lexicon in the last seven years I'm really happy to see that gone but that was what they were getting cold at that time um so that obviously tells you that we're going to be simulating a malicious attack right that's the kind of key freeze there so I'm gonna be a bad guy here interestingly you've got a wiki pedia now it's changed and it's changed kenick quite radically because now it's saying we're going to be assessing weaknesses and strengths and we're going to be enabling a full risk assessment so

whilst you're still talking about that about bad guys now we're kind of like talking more risk assessment more kinda like strengths and weaknesses so that's interesting to start with if we're still doing the same pen testing we were seven years ago we compete is to say that bro and I were doing the time of the wrong thing because it should be this year you can do nice color people about their stuff Preston our pen test the kurma gate last year and again you can see similar language around simulating attacks malicious Outsiders our own staffs were mentioning maybe it malicious insiders know as well but we're still simulating attacks right we're pretending to be bad guys rapid7

have a really long and wordy them a description but again simulate real-world attacks discover exploits were definitely exploiting things we're doing fantastic um and we're exploiting security vulnerabilities they're kind of key maybe because they make Metasploit you know they get really into the idea you're definitely exploiting stuff if you're gonna be doing fantastic so there's a lot of different definitions and also since since 2011 I do this first talk we've got this new thing red teaming red teaming isn't actually that new he started the US military many many years ago but in terms of the UK security testing industry red teaming is only really been kind of halt quite say for the last three or four years it's

really kind of come on if you look at definitions of red teaming they actually have very similar working they maze you know the difference I've seen that's not an article that talked about the difference of red teaming and fantastic and they defined red teamers as ninjas and pen testers as pirates so the idea being that the ninjas would be sneaky and get where they want to get out at the pen test the pirates would kind of wandering and break everything I don't know if I was that members of the red team are a bit writing and they thought I wasn't honestly but red teaming to an extent covers the same kind of ground you know

yes emulating a bad guy right I'm going and then we'll simulate about that I want to do things to your systems so red teaming sense a bit like pen testing or sounds like some definitions are pen testing other things happening since the last six seven years really the UK we've got the rise of bug bounties as well so again bug bounties aren't that new but how can one didn't start for 2012 but crowd 2012 maybe bug bounties make pen testing irrelevant I've definitely read quite a few articles that suggest exactly that you've got real-world attackers you four people sitting out there doing real-world attacks on your systems in the UK mother you're only paying them if

they find stuff so maybe that replaces pen testing all together is pen testing a dead concept hopefully not I'm gonna have a job um so what is a pen test made off if we actually bring this down as here right what actually is this pen testing stuff the first question every single definition just over there said we were going to simulate the attack of a bad guy which bad guy exactly are we gonna simulate are we gonna simulate hacktivists so we're going to turn up with a huge crew of people a lot of them all to be DDoS in your systems some of them or actually really know what they're doing and we'll be doing some

quite advanced stuff I well maybe we are but we'd have to be doing things like that so we're going to do DDoS and then using as a distraction to go and do something more serious that's why I hacktivists would do are we gonna do script kiddie stuff are we just gonna do automated attacks a lot of people in the internet if they're attacking real-world attackers they just run scripts right they can automated tool they run the automated tool against your systems and probably a couple of million other systems and if they attack the ones they actually get hold off will actually fail to the system so there's a really basic attacks if we want to

emulate those people we just need to use automated tools right so I could say I've done what an attacker would do if I'd done a pen test because I've just run some automated tools well if you're if it's script kiddies I'm emulating perhaps it's true what about corporate espionage what about you can a person who's gonna go in there and they're just going to try and steal specific data from your company and they're going to do it very quietly they just want to get certain intellectual property from you that's valuable we know there's a lot of people doing this stuff now and they're going to try and get out very quietly they're not gonna make any noise and

they're going to be very specific things because they just want something specific what about my describe Insider they've got totally different perspectives live access to your systems to begin with it may be good passwords they've got network access so you're disgruntled insider has got gonna be eluted hard well it's different in terms of emulating them I can't emulate them the same way because I've got half the knowledge they've got a war bar fabled apt the thing that happens every time someone gets broken into it's these guys it's a nation-state it was apt that did it that's what happened they obviously have got access to a huge quantity of stuff that most people don't have access to they can go

and get access to a library zero-day exploits so if I'm gonna emulate them effectively they also in a lot of cases can break national laws if I'm The Secret Service for a country I could generally break a lot of the laws of that country most pentesters can't break the law or shouldn't break the law and because you know you don't to go to jail you lose your job and what the coming is mine I get sued but you can't emulate these guys without doing that because that's kind of stuff they do so the very first problem we have with this or our definitions is they don't tell you which bad guy you're gonna actually and that's

a bit of a problem because it's hugely changes what you'd have to do so what box covers my test going to be what so one ways someone said to me was pen testing they described black box testing the way I tend to describe it I think of myself as standing I said a box with a series of sharp pointy sticks and my goal as a pen tester is to poke the sharp pointy stick into the box and hit a vulnerability that's exploitable how could a pen tester urs how many sticks you got and how sharp they are and how good you are at knowing where to poke to hit something right so that's white box

pen testing your scanning tools your sticks and your poking them and to see if you can hit something so a lot of people will say well black box testing why would I do that what I the box up and look inside right because surely at that point I don't have to worry anymore and I know at various points in my career I thought well surely white box testing is always going to be better and they actually turns out not to be necessarily the case the problem is when you open up the box you will see massive quantities of stuff there's huge con of code systems configurations and trying to work out which of these is

actually really exploitable it's quite difficult you get a bug in code because they're all codes the [ __ ] right do I know that's actually an exploitable bug that an attacker could actually do something with to compromise the system or is it just a bug in code that you know maybe it would cause a problem maybe it caused a crash maybe we'd do absolutely nothing the system will run quite normally so in reality white box testing maths it can be useful isn't always the best way of approaching it because what happens is you can get lost in the deck to the details and you don't know which these things is actually exploitable why not both and what cases

this is my preferred way of doing it give someone black box access but also give them access to the code the way that this works for me when I do it is if I'm doing a pen tester system I put with my sticks in and I hit something there's an error message and it says hey something odd happened and I could spend a lot of time fuzzing that trying different sticks and going which one of these just going to prove to me this is a exploitable issue or just a dodgy error message at that point it's really easy to open up and go wait a my stick though right it's hit this line of code

it's hit this module that's what the problem is either this is something serious and I need to report it or this is something I know that series and I can move on that's actually are useful way of doing it but again I'm I really am emulating a bad guy bad guys can't do that most of the time most of our definitions of bad guys don't get to open the box up and go oh look but pen testing is actually the most effective way to pen test it might be to do that what's up school this is if anyone's been in the pen testing industry at all scope is always a fun one is the school just an application

all right so maybe I've just hit in Ashley encases if you kind of imagine the sole of the left shoe there that's my school I've been told to test like five pages of one lap one on an app that is my school no testing anything I would say to that Robert is a school the systems that make up the network or make up the application is it maybe your entire external perimeter or is it everything to do with your entire organization there's a huge different variety of scope there now again we're wanting effective pen tests most people would say well obviously the answer is attackers will hit any of that so you mister pentester share their an eva at which

point if I'm a seller I go great because increased scope equals increased cost costs money the more you increase the scope of a test more you've got to pay for it realistic we speaking if I'm a buyer of pen testing I probably got budget either the project I'm working for has got a budget or my department has got a budget um I can't just say to you hey mr. tester just go out and blitz absolutely everything in my environment because it would take it would cost a fortune because realistically speaking it costs money so scope or pea tests but again if we're emulating bad guys we should only really be ever doing this stuff that cause bad guys most cases

will do everything they aren't just going to look at the fact you've got this kind of thing on the bottom of your left sole and say that's the only point where I can actually hit you um so scope is a bit of a problem and scope limitations and on any pen testers in the right have you ever had a situation where they've said no testing before or after a certain time I know I have within visitor or indeed sometimes with business hours you're not let's talk to our systems and lacks it's between 2 a.m. in the morning and a.m. in the morning which is no fun um alternatively a little fragile vias over there just to demonstrate I'd be told don't

touch that system don't trust our system you can touch anything else in the network but not laughing at guys typically don't care they don't care about what time of day is and they definitely don't care about your fragile systems um so if we're gonna be like bad guys this doesn't make a lot of sense but now I'll flip this over my head I'll say right I'm know that internal security person in a large corpora um and the business says to me this system here is really heavily loaded and we're worried is gonna basically break if anyone does anything to it please don't touch on the test well I don't want I say right don't touch that system because I can't risk a

production outage in the middle of our business day for an important system because businesses don't like it when their systems go down in the middle of a business day they're not happy about it so you have to have scope limitations if you're in that position you can't necessarily say to the tester do it everyone um this is a question I saw asked on the internet let's go he's the personal email of staff ever in scope of a test I sure has been when in terms of Penta shoot Pantera's be allowed to go and send malicious attachments to the personal email of staff and a company I don't want to say yes yes and was he yes yeah okay I don't

see no so they asked this question of mine and our survey said this is that's why red teaming more people said yes than no so no one's ever asked me to do that and that's kind of lucky because if they did my very first call will be to a lawyer because I'm fairly sure in the UK that's gonna be illegal computer misuse Act I don't have authorization to touch Gmail Google have not authorized my test the parks me his home PC I may be touching hasn't authorized my test so clearly suicides not gonna have a good time of it these two Protection Act I'm processing data for person who's not the entity who I'm contracted with I'm

potentially gonna go to jail if I do that however bad guys are absolutely going to do that bad guys that's one of the main ways they're gonna get in they'll target the email of the CEO because he's got the board papers on his PC don't target the email of the child of the CEO because he they use the same PC as the CEO and that's where the board papers are will go after a non-executive director who doesn't even work for the company because guess what he's got the board papers to I'm depending on the type of attacker so this is one of the problems we've run into with pen testers and emulating the bad guys is that's

illegal or probably leave I definitely wouldn't touch that with a very well worded letter from a lawyer because I yeah I'm no lawyer but I wouldn't try it without a lawyer for sure so that's a very bitter a problem school so the next question is where do I test what environment do I test it now the obvious answer here is production right the only environment where you can really test a system is in its production environment it seems like a really easy answer doesn't it well here are some ways I've broken production systems while testing them I have knocked over a web application and another web application in the same system by putting a single cool

character into a login box and pressing enter I have denial of service a major e-commerce site within map I have the credit card processing for another set company within map and I once got our call for my company name months after I've done a pen test saying the test data I had left on their system cause they're not going to fill there are real risks was test in production if you test in production you've run the risk of crashing the system even the most non-invasive test nmap is probably considered to be one of the least invasive things you could do to a system I have personally crashed as I've said at least two sets of systems where

they're alone nothing else didn't do anything else to them done touch anything can do it and that's one of the things when probably pen tester you've got to pay attention and a lot of time when you're selling one of the questions of custom all say to you is is there any risk that mice that anything bad could happen to you when you do this pen test some pen testers will see or no no I don't think any real pen tester would say no there's no risk it definitely shouldn't if I know it says to you if you didn't ever have power harvester you're in a meeting you see if someone's any residents gonna Peck my system they say no and it's

anything other than a configuration review I would be very dubious the statement they're like broken things I'm not just I broken things lots of other people have broken things in the most amazing ways I've heard of someone wants boots on the ping they paint the system it didn't like it that much that wasn't a happy the thing failed though it wasn't playing ball you can crash a system with anything so my personal opinion here is for a lot of tests application tests definitely don't do them production don't ever test me blush because if you give me a nice test system and you give me a free rein on it you see no one else is using the system

so please don't do more tests in the same time as I'm pen testing because it was for everything down and when it doesn't work they're wonderful is there more it was me um I can go far harder I can use techniques I wouldn't risk doing on a production system I can't because if I'm testing some high value production system realistically speaking I want to still be on the job at the end of it right if I sold you the system it's my job I personally have had when I was a seller of pen testing I had people not pay us because the Kristin fell down on the first day of testing they didn't get back up again

that can happen all right so I'm obviously going to be a bit careful to make sure that doesn't happen so a tester can't be as hard and as food as they would like to be in a production environment so test and test environment don't test in production so next fantastic is pentesting an art or science if you're a hacker if you were emulating the bad guys obese it's more like an arm right you're going in you're doing some things something might occur to you it's great and strikes you dive in you type some lines of code in the way you go when I was an internal security person for large companies they would say to me hang on we've had this

test done last year and this year by the same company we get different results what's going on and they'd want to know why it was different and that way it's kind of hard to see what it's kind of an arm you know it's you know because it's being sold as being quite a kind of scientific concept but in reality the real answer to this is is it is it depends kind of on a test type it also depends part of it should be a science right if you do the same every week we do the same basic stuff to find the same basic issues but there's always going to be that element of the experience of the

texture sitting in front of a computer and going oh I did this because I've seen this five systems ago or ever this blog article last week you know or someone on a mailing list I saw said something that just triggers a thought in my head and I couldn't type it so but a lot of people want it to be a science and and there's some good arguments for why I should be a science but in reality is not it's never good I don't think ever will be but maybe these people more sciency so this question comes up people say to you what methodology do you follow as a pen tester the problem is there aren't really great published

answers for this a lot of people will see Peters pen test execution standard which was good but I looked at the wiki Lydia hasn't updated in four or five years there's been a lot of new tools and techniques in the last four or five years so someone says do you we just use P tez I'll be there have they been keeping up to date they might say oh well top 10 is also plastic someone says oh we follow or top 10 or top 10 is not a pen testing methodology it never has been and it really never will be its list of the 10 most serious risks in web applications at any given time and the latest version has got

actually it's got a new entry which is failure to do appropriate logging and monitoring so if a pen tester tells you they're going to test the oil stock 10 challenge them on how are you testing my locking in monitoring as part of your pen test because if you aren't doing that then you're not doing but of a problem so methodologies through useful to make sure everyone covers the same basic things so most tested company should have one but they're not going to get you one percent of the way because then we'd all be robots and there's a question as you're told about could you ever completely automate texting and I don't think you can because of that inspiration piece

and if he alright um I was thinking when you said that if I ever look good enough to do pen testing if we do enough trace if your code in the first place so once the I write secure code then the AI can do the fantastic because it we have to do both sides the tank but they're very similar kind of things so what is my report look like is obviously a good question if we went to the pen test it if I'm a bad guy bike guys breaking right they exploit some stuff they get the data they're trying to get and they go away the report would say that it would say we broken we used these

exploits to break in on these systems got this data when we left when I was one for internal corporates if I presented a report someone they would say ok if we fix this is our system gonna be reasonably secure if all you've got how they broke head what is thirteen other ways to break in right they found one broke and went away so the reality is most pen test reports shouldn't look like that they should look like that if you're doing real adversarial testing but most people want what they actually want to see is a list of vulnerabilities and systems you're looking at they want to know how to fix them and they want to

know that if they do these things they should be in a better place a hopefully a good place at the end of it all so the report it kind of matters what is that gonna end up looking like um and it might not look like what hacker butchers who does my pen testing another controversial question does my pen tester have to have a certificate after their name mascot lose by the way there were warrants big stickers that some they're rare and very valued commodity that's yeah some these desk was this day so um there's a controversial question pen testing which is should all my pen testers have certificates have certifications right um there's different points of view so one

point of view says the problem of certification is that no certificate mirrors real-world experience which is true and certificates are an expensive process that can exclude people for to do it this is also true third aspect is certifications exclude people who don't test well right tests generally involve a high pressure sit down it's not really like a test really is and some people who can be brilliant testers don't test well and you're excluding these people if you start requiring certificates the problem with this is the other option would be should they be hackers right so obviously we're saying well you emulate the bad guys who better to emulate a bad guy than a bad guy right somebody's done the stuff that

you're trying to do in the world the problem here is go back to my life as a buyer I don't have time to interview every pen tester he's gonna do a visa work for me right I don't have time to assess personally there are qualifications but I want to make sure is they've got some sort of basic idea of what they're meant to be doing and they've been through process they've got a good base level of qualifications understanding you can't do all this in sales materials because every test company's going to tell you they're brilliant man obviously you know I've sold stuff I'm not good to say yeah I'm not good at that sort of thing no one

makes sales doing that right tell you that for nothing so what certifications do is they can provide us a base level they can say okay if you're getting someone in you can assume this we've got this certificate they probably know this kind of stuff reasonably well you need to watch and make sure that it's for the right type of testing so obviously if I'm going to get a crest webapp test right you say you want some CCT at OCCC Inc because those don't cover the same things um so certification is a tricky one again if we're gonna be like bad guys then maybe we have more people who with it with a bad guy background but the

risk for that is I don't know how a buyer and I'm a small boy am I gonna buy a pair test how am I going to assess whether you're a good hacker or not like a list of people you've broken into that's not something else gonna put on a CV is it all right if I broken into all these companies not harming hard for you to assess who to buy from um what other thing to clean up really really annoys me pan testing is not auditing right I found what I've got in the difference and the difference to be clear is in a pen test I have no personal investment whether you pass or fail my report

doesn't say ever say pass or fail I don't issue a pen test reports as you pass the order you failed the order whereas auditors that's what they do they issue a red report saying you failed or a green report saying you passed or sometimes an amber report in the so as a pass/fail thank Pam testing isn't I've had people when I go to do pen tests I do things like saying oh we turn the system off and they'll tell me they've turned it off yeah you're still gonna be vulnerable when I leave so you know you musta left it on I've had people say or we used to have our passwords on that wait what would wait

them off while you were coming in I don't care I'm not an auditor but people think your auditor and the risk is they then won't tell you anything so they'll say oh I'm not telling you anything who's you the auditor because if you worked in certain industries the very first thing you could told was don't our auditors you'd never talk to others so it's a bit of risk but pen testing is very much no matter what it might be and that's a debatable point it's not auditing so why is all this a problem good idea of course so what see here is the definitions we've got really don't tie up with the reality of what gets

done that's a bit tight to me this is an issue so I'll give you a kind of an illustration the promise everything as a pen test or azam bees are millions of different zombies there they're all pen tests so I've seen people sell automated vulnerability scanning as a pen test I've seen them sell manual vulnerability analysis a pen test I've seen them sell code reviews a pen test I've seen them sell social engineer escape and test and I've seen even seen things like pen test as a service so time automated pen test I've seen a legitimate argument an article written with my quite senior person in the industry extolling the idea of pen testing as a service but how

can you have this fully automated thing and emulate this real attacker who's a human with brains and definitely experience so the problem is if everything is pen testing nothing's pen testing right and we don't have a good way of defining it it causes confusion and this is basically the problem so say I am a buyer right I'm our a mid-level manufacturing company and I want to buy a pen test I'm going into e-commerce for the first time so I've created a website and I've got it I've done got agency Dre my website for me I'm going to process credit card transactions so my bank or my director summer I said here you no need to go and have a pen test on this

thing right I go to the market and contact three companies come here come baby come me see I say to them I would like a pen test on my new system company it comes back to me it says great excellent see me an email quote is a despot 2,000 pounds my gosh your pen test Company B they've phone call with me first this and then they send me an art of 2pj for that five thousand pounds Company C comes in to come out of their meeting they do a workshop with you they come back with a big wall see document tells me fifteen thousand pounds which one of these are my gonna buy they're all pen tests

they're all pair test companies they all describe themselves as pen testers and they'll tell us how great they are problem is let's be honest if I'm a commercial person who has been me being told by a bank today I'm going to do is I'm going to go to company a company it cost me two thousand ends and that's well within my budget maybe I've only got three or four thousand pounds I'm I definitely can't afford Company C because they're 15 doesn't mean that I didn't know that's how much it cost so what company is gonna get bought a lot because they're all the same they're all telling me the same thing but they're telling me four very

different I meant some money so that's not good because what they came here probably flip that round I'm company Z I've just gone in I spend all the time with his customer i sat there and I've had a workshop I've really understood the requirements I've gone away I'm taking the time to write up this document proposal for them fifteen grand and I call the guy up the next week and he says sorry we've gone with another company there were seven times cheaper than you well I just wasted all that my time and money trying to sell that job personally as a real-world thing I've had that experience when I was selling fantastic I have a situation when I went in expect

the job out did the job said it was it was 13 and then they come in and said you were three times more expensive than the neck then the board bid we've gone man and I and I looked at that when I literally can't work out how they did that job in the time because it was quite a complex job it happens that we were both selling fantastic same thing so it's like well I've got can cover myself tick box or my check box compliance test I'm gonna become be so company sees got a real problem cuz I can't justify that it's gonna have real problems just to find the cost and what that everything does is cause Company C

to start selling see cheaper tests so they actually win the business because realistically you're a consultancy you gotta sell stuff right so you gotta have to go with Mark because causes compliance confusion as well of everything's a pen test when I was in various companies I would try and get people to put into their contracts when they were buying things you need to have a pen test before you buy this product or are you at the parties to give you some assurance and I didn't have like two pages of the contracts to be able to tell him exactly why men I can only use like two lanes that's the kind of space they'd give me

so I just put in pen test doesn't mean the only words I had but of course that could lead to the problem you have core which is if everything's a pen test you know I need to see some of the reports that come out afterward so you're like that's not a pen test so it causes compliance confusion if you look at standards that PCI they'll say you have to have a pen test but we've seen that it's a bit difficult to what pen test is so he's had compliance bit of an issue and career confusion a bit of a problem if your are a pen tester so I'm I'm a pen tester you could have people who do

probability scanning an analysis they could legitimately see I'm a fractional pen tester you can as someone else who does scatter and deep scatter stuff they're also professional pen tester and you know someone who reads Java code for a living they're also professional pen tester they'll do things are swords pen tests but they have a very different career progression is just kind of difficult in the industry to say you don't have different trends so ever just give me a pen tester so how could this be approved it's obviously broken and we're trying to kill it off finally get that zombie and blow his head off how could we do it um I'm gonna try and use a model they're using the P word so

I'm knowledge C pen tester for the rest of the presentation after this slide let's see whether I could do it it's really difficult um almost exactly what I will same principles the first thing you need to do so if you're a company you're looking to get pen tests the very first thing you need to do is which of these bad ah excuse estimates skew assessments you need a security assessments and you begin my assessment of threats so which of these bad guys do I need to worry about because not everyone needs to worry about all the bad guys you won't need to worry about the script kiddies you have any system anywhere you need to worry about any that attacks if

you're exposed to the Internet and you probably are these days you're going to get attacked simple as that but not everyone needs to worry about the NSA realistically speaking so not everyone needs that very high ever assurance at the end of the day um when you do security assessments I would always if you're starting company start with lusco white box find the easy stuff first do not pay a high-end red team to come in and find you got em SOA or six seven one of the most naughty vulnerabilities in the universe they waste if you can't hire em but if you pay you 40 50 thousand pounds for ready come in and they break came using

that you just wasted that money because they've done their job and you you know you've not given a hard time you need to get rid of this easy stuff first gradually increase in scope and for my own personal opinion only move towards adversarial testing this kind of black box I'm actually gonna be the bad guy once you've got all the easy stuff done though you know white box stuff out of the way and testing is only one part of any security program this is critically important the number of times I would come along to a project and I do security review on at the end mr. Floyd the work and they would say I think

things that were like architectural issues so I find something which was clearly should have been thought out when they designed the system and not when they were implementing it and I and they were going over to happen today sir is everything okay for us ah bit of a bad time if you wait till the end that just do your security testing in the end you're gonna have a bit of a problem with it so start off step one on the six step program to do in your security system properly we're going to do your ability scanning everyone should you vulnerability scanning everyone should assess their systems all of their systems actually there's a step zero the

star of this which is asset inventory very unsexy if you don't know where your systems are and you can't enumerate them all you could absolutely no chance because you can't scan them and I'd be surprised I mean companies do know that um what about scaring everyone to different ways Gary it should be done white box there's no point in trying to hide things from your own ability scanner give it credentials let it login it will find far more stuff in all these different done vulnerability scanning a new black box it just for Allah probes get stuff back and Kenny gases I'm kind of guessing whether you've got stuff give it credentials it'll log in and go

you could freshen this of this program that's got vulnerability there's your problem you know it's a definite always do white box start off with easy stuff external systems move toward school coverage do it in production obviously can't really not do vulnerability scanning in production report everything report all the vulnerabilities and this one you maybe don't even need to get a test company involved to be honest with you internal resource can do this in a lot of cases it's going to come down to individual companies to who work makes more sense for some people make sense Tess come here at this point some people you can get yourself there are open-source scanning tools like open VARs which obviously free or Nexus

licences are too expensive once you've done that you want to move on and get human involved how to get kind of brain about going on behind there differentiation between variability and assets and scanning is when you're doing scanning a tool will say to you hey I found an open file here what it won't tell you is the difference between an open file share with some tools that are meant to be publicly available and I wrote the file share with HR day today right it doesn't know because it's a scanner to some files in there I don't know whether they're good or bad it's not going to tell you whether it's someone salaries which is deeply

confidential information or not you need a human to do that so again everyone is to do white box same sort of scope set up cut it in production report everything and this point it can start at the point where having a professional who kind of looks at this stuff for a living makes sense so it should be cheaper obviously because you're just going to do the analysis based on to output but test company might make sense or you might want to do a time then you can move on to doing manual security assessments everyone with custom systems everyone who gets code written for them will need you need to counter this stuff manually scanning tools one of the dark secrets

of scanning tools is they don't tend to work very well on completely custom systems webapp scanners will try to do things but they're never perfect and they're really noisy and break-ins but wait this is the assessments I would do both ways around so I'd say give someone access to the external system but also give them the source tool because they're going to have a good time of it that way ok you were going to get better results custom apps test impossible do not do this in production because it means that your tester could be far more robust in what they do without worrying about knocking over your whatever system it is and bringing your company to a screaming

halt report everything and probably done by test oh the flip side of this is configuration review is this one gets kind of a bad press because it's really boring work I've in a lot of cases but it's really necessary when you it's not yeah not too horrible but they can be kinda boring I mean but they are really really you so what I mean by configuration reviews is when you buy a stopper product you buy a system from someone you take it out of the box you plug it in was the configuration good bad or indifferent in the security standpoint a lot of vendors will make it so that the system works quickly and easily they won't

necessarily want to harden it this hardening breaks things and they don't want to break things they want you to have a nice experience with them and not to send it back to them right so they're going to make it as easy and working as possible so you need to do hardening the other thing is over time configuration slips the classic this is firewalls if anyone's ever managed a firewall go back after five years of it running and look for all the rules that are redundant and don't apply anymore the ones that say temp next the more tests or I'm going to remove this after John's done his work that sort of stuff if you've got comments which must attend

they don't have comments on them and you just have to guess so configuration reviews are important if you've got anything that needs big systems and configured firewalls is the absolute classic Rooter is that kind of thing obviously have to do white box and this is where I'm talking about things being totally not like um bad guys no bad guys going to do conflict with you for you probably or if they are you in a really bad position but the conflict reviews are the best way to assess this stuff to assess a firewall rule base it's work out whether it's doing his job properly the absolute best way to do it is open it up get the config and analyzer not to

try and probe it from the outside world fire port scanning a firewall that generally fruitless task so don't do that black box anything with custom configuration god of production doesn't make any sense otherwise and generally also the reason to reproduction is you score where the things creep over times where every time I practice systems been in for a long time it tends to accumulate croft from manual changes because something broke something didn't work and someone makes a change and then it gets left there because no one want to take it back against it might break up even after the problems fixed that's in happen all this in test environments so or a different set of things happens

a testing problem so you have to dissing production but it should be risk-free the own type of if I was it to me is any testing risk free I'd say config reviews unless the tester something's silly they're not going to break something in the bill of a conflict review that's kind of hard general is offline as well report everything and you probably want a bit of external expertise for this one because knowing about firewall reviews about a specialist skill then red teaming we get onto the proper adversary so I would stop using the P word for adversarial testing because I've said the problem is everything's the P works and nothing is so let's start calling it

red teaming right I'm gonna say we want to do adversarial testing or I'll call it red teaming not everyone needs this the only thing I've seen some again some proponents of red teaming they seem to imply everyone should have red team's I don't really think that's necessarily true I think for a lot of people if you did all the other stuff well you'd be doing okay but if you're a high-value target you're gonna be the target more serious people so when you do your analysis and you say hang on we've got intellectual property relating to multi-billion dollar deals then you need one of these because the attackers in multi-billion that for multi-billion dollar deals will absolutely use the

kind of stuff red teamers will use boxcars gonna be black this is where we're actually going to do proper adversarial testing we're going to be the bad guys as white as possible do not constrain your adversarial testers artificially otherwise you're not getting a proper test but recognize the boundaries that are there they can't break the law they can't go on to a marketplace and buy zero days probably must want to get some funds for that I've never had funds for that I would last look at this table someone's yes they might get funds for going on to market sounds like sending thousands of Pennell zero I'd love to see a test actually did that but I've never seen so

yeah you can get your own hopefully you book it on to start off either tools that do say my zero days or your you come in yourself Scott set of tools that do you know non-public exploits so that makes it better definitely the other option I've seen here is for I've seen some red teams where they have what the zero day card so this do is they kind of go through the red team and they say any one point that one is eluded a card they can play and they say we need you to pretend that we had a zero day for the system and give us the access that this would give us but you only get one card per test so

the idea is that simulates the effect of one zero day which is not bad now maybe if you're really hired gonna do a production nor the place to do it I know they should only reporting what they broke if you're doing a red team and the red team are start telling you here you had SSL v to enable them to your servers to me that's not a red team all right that's a security review red teamers should be tell you what they broke how did they break in what did it could to get to was a sensitivity of that data you know general things should you take from this and also did you'd find them there's

obviously your defensive people should be finding these guys as they do it it should be a proper adversarial thing I'm done by experts this isn't something that there are a few the biggest of the big companies sure you've got an internal red team or anyone else does not internal rating because they're really expensive and to go along with that if you've got any significant internet-facing system or unique software I'd say have up up empty why because it's cheap owner abilities sure it's not cost-free bug bounty programs do cost money to run they're not free to run and you'll get a lot of dross I'm sure you've seen that you get a lot of drops in his programs people reporting

the fact you're missing a header and wanting paid for it right and you have to do with those reports so there is an overhead to running them but on the other hand they're really cheap source of vulnerabilities on your external systems compared to having and they happen all the time so you don't have to say I'm gonna schedule this for next week just open up writing you were getting hit within minutes it's obviously black box realistic and this is nothing where I've seen lots of articles that say hey what bounty programs can replace pentesting no they can't because people in again two three strikes on my stop talk right I said they say it can replace all security

testing with bug bounty you can't do that what babies don't get given I've been credits look at the admin portal what bounties don't get given access to your internal systems so they do in town that's it Wow that Google someone okay well unless you're someone really that's interesting so most bug bounties you wouldn't if you said to your IT director I want to give user creds to hold a random people in the internet for various companies not many IT directors apartment there may have by such serious people maybe that make sense them but the vast majority people it doesn't make sense to give random people from any country in the world access to your network because

they probably do really bad things to you so except that bug bite is generally on the externally facing systems or software you sell which is the other option ah a production what do they break the only ever tell you what they broke that's the Dems in as well they're only bug bounties will obviously focus on what you pay for so if you don't pay for a certain class of vulnerability you don't pay well for it don't expect to get reports all right because why would think there are they're there for money they have to make money from you or or kudos or whatever else you're selling I've done but but buncher's solutions to all this

so that's a program that goes through without mentioning the P word for two times the problem is that it's very badly to find it's all things to all people and as a result of being all things to all people it's useless as a term we would be far better to get rid of it as a term and start using different terms because it is useless and it's damaging the industry now because we can't have a sensible discussion of its scope because everyone's selling that I can't say pen testing now because I'm mad about it so I everyone selling fantastic so if they all sell pen testing new and selling pen testing if you can't compare like with

like as a buyer you can't compare it as a seller because you can't differentiate your services easily I'm selling pen testing he's selling pen testing he's cheaper than me why is that it is perfectly possible to have a model that doesn't use it so it can be done there we go all the different definitions of vulnerability analysis manual security assessment of red teaming that we can use without having to use it so it can we can I think we can get rid of it or we should be able to get rid only thing you can't quickly get rid of it we should definitely look to move towards that questions I think I haven't read it

recently the challenge I had with PCR in the past is the use CVS s as a measure of possibility and CBS s if there was no CBS s is a vulnerability scoring system I really hate CBS s with a passion and the reason I do is because it gives an illusion of a metric it lets people think that you can actually rate these vulnerabilities the good yeah so I think the problem is that people did use the option C CB SS because if you actually have my CVS test scores you find out that it's a guess times I guess times I guess yeah I think I think it's good I mean the thing is will drag on PC I and I actually like PC

as a concept because it's got a hold of companies to start looking in security who literally never would have otherwise literally they would have never touched it you know unless poked really hard and made to do it so I like from the app side that is good to see they're developing I think the thing I'm a PCI is they've kind of stuck with it you know there's a lot people kind of pushback on PC I tell you let's get rid of this in the first couple years of implementation when it was really horrible and it's nice to see there actually kind of iterating on not giving up and saying hey maybe we can make this

better other than just giving up so yeah having something like that but you say it does tend up ended with amazing I'm not sure there is a perfect answer to compliance and pentest because I don't think it's always going to be a judgment call and that's one of the problems I have with pen tests where people say the pen test is a pass/fail it's like well define pass/fail with this concept of the test I've just done compared to your overall environment why tend to do is I try to say to people do you have a representative test environment and when I'm an outside consultant I can only go so far to that say look you know if you

tell me it's representative I'm gonna take its representative if you tell me it's got certain differences so you can see me for example the SSL certificate on this on the test site isn't right and I said no problem I'll talk about the test port you know I make an informational note saying I didn't cover any of the usual SSL feelings because you've told me that's different so as long as everyone understands what the difference is between the environment R & R you can actually have a test skill that makes sense what don't MC is app testing it's much easier to do because people should have test environments for a MOOC for their own custom apps you would hope these

days and in a way some of the modern practices like DevOps make this a lot easier because their scripting entire environments so usually you should see one that terraform script which builds something in the cloud run the same thing for test is they run production so you can say well as long as you run from the same script you know you've got something that's representative so it does depend and ultimately you kind of have to go back to system order and go do you have something which you're comfortable with if I give you a finding on this you're not just going to turn into Toby oh it's not on that production so you kind of have to go back to system

owner and let them manage that problem and say how comfortable they are and some people are more more or less I does this gave me sometimes I'll see a big system they'll say we don't have a testing member well how do you know before you pushing you version it's not going to break everything because you never you know she looked at systems so yeah it's a trick you'll not just tell me it's not mobility I've also talked have them tell me things like it's not a high risk because a lot of people unfortunately and this is anything about pen testing auditing they'll say oh we're gonna fix everything that's raised high and we're not gonna we're gonna maybe think of a

fixing stuff that's medium and then there's a little forget about it we're not touching won't be there when you company next year so they'll trying get me to decorate in from the ability my approach to pushing that back is I say look this is a technical risk assessment of just this system you may have other controls in place but I didn't assess those I didn't test your other controls are you telling me oh it's mitigated because we've got this and that and this and that that wasn't my school if you want to add that to my scope then won't talk but I can will tell you technically what I've done you either then have to fall back

on something like if it's a fast run ability so I had one company who said to me Oh our Oracle is unsupported we have an Oracle database it's entirely unsupported it's missing tons of is open or closed good with security sure that's okay all right no no really isn't but that was easy look at I had loads of ones with the Oracle himself submitted as a high so that's a TVA you feel like a capo you disable the vendor says as a high such a high are you changing map if it's custom code I have a low boost comes I've work for you is a high medium low and you kind of define high and you say well

highs things like anything which compromise the system completely on its own that's a high and I'm not gonna debate there are some times where someone can say by the label you know because of this we things a bit lower and you move a little bit but you don't go I would never write over nobility off and say no it's not an issue I would just say it's for you I would say this process it's a defined if you want to do that but my my all my report is is tell you what Ted or risks were I mean if you kind of keep it to that discussion hopefully that stops them from being quite so what's not more

time though because it's a the some people just want to clean up or other people do want to test properly something's want to clean the world I haven't by new people who have it's not yeah it does happen and you can find compromise systems as part of as part of what you're doing I have had ones where I'm I had to say to someone you need to go and do an instant response note because if this vulnerability hasn't already been exploited I'll be extremely surprised so I didn't necessarily phone the export you're having phone because it told me I've had one where I found evidence of a critical vulnerability and I said you want me to exploit it so I

can tell you what the thing is this is no no don't touch it it was but option and I said look you need now to go and check that sister because I'm going to be really surprised if I'm the first person to find this this high value system at this was an easy vulnerability so yeah it does happen and that's never fun yeah well the dirac they make conversation is really funny and if anyone doesn't not in the industry the funny thing was before I did I T the security I was a I think itself back in the nineties my dear it back in the nineties because that company sought me out at was a thousand pounds a day

that's pretty much the benchmark for security testing not but in jet on an average no 18 years later so in all that says we got a real pen test skill shortage well we had a little skill shortage surely if demand is higher than supply rates go up doesn't seem to happen so that's an interesting point the problem is that if anyone can sell fantastic literally anyone can sell website they can say I'm an expert at pentester they buy a couple of tools and they can sell themselves as a pen tester without any certification knowledge anything and there's nothing to stop you doing that legally that's the problem so that's where you come back to the

certification argument you could say well well everyone should be certified because if everyone was certified we were more like a accountants you have a body that said you can't repent test but then you're heading into a very structured slow world that might not suit the rest of IT the rest of my team is far faster than that and if everyone if you had something where oh my methodology told me that this was every three years - that's what the methodology board meets and me'll boss the IT guys have gone and done something wacky so I think it's maybe just a correction of the professional bodies have to keep doing their job let's do a great job of trying to articulate the

benefit of doing testing properly and using companies who you know go through your needs properly they just have to keep doing that government I think could have more virile they could have more overall or actually saying hey you know there are differences here and we're gonna give you you know they're a condition of these differences I'm not sure how easy it's gonna be to get rid of the problem people selling stuff I really cheaply and calling up and test parchment the other thing is education educating customers as much as possible to say hey be aware of what the reference is someone sells something is 10 times cheaper there's a reason well factor you