Rory McCune - Penetration Testing Must Die

i'm this is basically a rant and i was i will admit i was kind of surprised i got accepted for this um i put it in when they were like presentations and they were saying look please people put more presentations i think that's good i'll just go hang on but no something i actually have to talk about this so introduction the obligatory production slide who am i um i team information security last 15 years now um i'm currently director of selling elements and we have another product over there um and uh scottish check release for awards and there's my email and twitter so what do you want to talk about why it must die right

so pinterest has to die it really does and i'm going to start off with why it's not to die and i've got three reasons i'm going to go through and because i didn't want to die and have nothing left afterwards because i'd be out of a job um i'm going to suggest that we actually do something to try and fix it so that after it's dead we can actually have something in place and pen testers can also have work to do so what's in a name um the first thing to do is just and this is one of the problems i'm going to start off with is to talk about what actually is pen testing so i did

the obvious thing when you do this i went to wikipedia i said wikipedia what is pentax and it told me this penetration test occasionally dentist is a method of evaluating the security of a computer system or network by simulating malicious crackers anymore but you know apparently so so so what does that actually mean okay it means black box bad guys don't typically know that or you hope anyway know the ins and outs of your organization most attackers numerically are coming from an external perspective so they don't work with the full schematics of your system sitting in front of them so it's black box what else is it goal we're just trying to compromise right if you ever get a test report and it

says you have sslv2 enabling your web server that wasn't a pen test because pen to hackers don't care they don't care if you guys sleep to enable because no one's ever exploited it as far as i'm aware in the life i'm happy to be contradicted but i put on lots of reports but i never see anyone do it it's gonna be realistic it's got to mimic the real thing that's important i'm trying to emulate real thing but my first question which real thing am i talking about am i talking about an automated bot that was driving the internet doing sql injection on people am i talking about anonymous because you've annoyed them am i talking about

a foreign state am i talking about organized crime one of the problems with pen testing is which of these things are all of these things is it meant to emulate it's kind of in a lot of cases unclear so overloaded terminology everything's a pen test everything you ever get sold from a security testing company in a lot of cases has been described by someone as a penetration test so they will say i'm going to build a scanning is a pen test i used to work in banks and i'd read pentagon reports and i'd see what was blindingly obviously necessary but the executive summit told me it was a pentest that contest i'd say not where application security

assessment this is far more common you actually want an assessment of your application and what you get sold is a pen test or what's called a pen test so when someone's buying it how do they know what they're getting because everyone's going to call it the same thing they're going to call that contest code reduce pen testing might happen yeah anything else uh basically anything you've ever been sold by securities consultants none of them so clients aren't ready for pen testing that's why i'm definitely sick why aren't they ready let's go what is the whole thing okay so we've said we're going to limit attackers we're going to do what the real guys have done

any time you do a pen test any pen testers people do testing in the audience okay have you ever had a scope that said you can do anything you want anytime you want you can crash systems you can do anything at all you like yes good you actually have done a real pen test in most cases that's good because in most cases you you've seen someone is it okay if we you know we're doing these buffer overs that we might crash your servers a lot of companies and clients will say no no no please don't crash our servers because you do that the business are going to come in they're going to hit us for having crashed the service we're

going to get kicked off site and i've seen pen testers and i've been on the receiving end i've been told to stop testing while i was port scanning something because i've affected their their internet facing firewall reaction is not just that because they're not confident that they're going to break this ever online now that is yeah well this is it it comes down to confidence a penetration test is designed to prove or i think it should be designed to prove that you are secure as you think you are not to assess how secure you actually are so if someone doesn't know how secure they are well then they're they're not ready for a pen test if you get someone saying at

the end of the report you had the report you said yes i broke it so that's a goal-based test i broke in and here's the systems i compromised and here's the data i got and they say great if we fix these things you broke in by are we secure well they didn't want a pen test because you said well actually no i didn't look for anything else because i broke in i thought what an attacker would do they wouldn't go and find all the other 15 other ways they might be able to break in but a lot of customers what they actually want is assessment to say are all the obvious things on on this

system on this environment secure have they been fixed how we fixed the major the majority of them that's what they're looking for because they want to say if we fix those then we're in a good place if they've got that far and they think they're okay then they're ready for a pen test because then you can actually go in and say can i compromise you you don't think it's possible and that's when you get your blank scope do you really think business wants to attend this because they want to see whether you can break it now you should do it because the policy says yeah absolutely well that's one of the problems pcri is a great example pci in

a lot of ways has a lot of good to it because it's made people think about security and they never would have if they hadn't had pci there's so many companies who wouldn't touch security it wasn't pci however one of the terrible things it does is it says you must have a pen test and it uses those words and that's what i'm saying overloaded terminology because there's no other word term for them to use they had to use pen test so they used pen test and as a result everyone's now going here and saying well i have to have a pencil can't kind of i can't buy anything else from you because i've got to pass pci pci says i

have to have a pen test that's a real problem because it means people are selling their own thing and they're buying the wrong thing and people are also not delivering the stuff they want to deliver so yeah is exploitation okay are there no spoken limitations are there no abundance posts do you ever be told i've been told in texts anymore because hackers aren't going to say actually you're right that serves a bit overloaded today i won't touch it it's not going to happen so and here's the bit that's slightly interesting mission impossible i'd argue that even in the case where you get that blank scope and you just get told here's two weeks do whatever you like it's still pretty

much impossible to accurately mimic mimic current high-end attackers here's why feel free to argue where's the data so these days all about data you're trying to secure the data where is the data is it in the company that you're actually working with or is it in the third party it's the third party in scope your test is okay if i go and bounce to your outsourced code developer and try hacking there and then coming through that i paid this note because i bet you they're contracts in the list of that cloud service provider in school is okay by trying to hack amazon i think amazon's going to sign off on that ever for some client there to try and

penetrate their own systems they'll let you work but what school would they let you attack their admins on pc yeah that's good and that's heading in the right direction that's the next method fairly obviously as a pentester you're not allowed to break the law or it's not a very good idea if you want to remain working in pen testing so spearfishing is probably okay these days some people will start buying into the idea when you look at rsa how are they compromised from what we've been told email with an attachment however commodore compromised email with an attachment to see a theme here right now the majority of tests i'd love to see this happen more don't allow

client-side attacks so you're not allowed to start emailing random people in the organization and again it's the same reason if you work for a large organization ceo phones up and says hi my machine just popped up saying it's got a virus or do you know anything about this you probably don't want to say yes yes actually i commissioned some guys to try and break into your pc that just wouldn't end well in a lot of large organizations so but again hackers don't care home email can i can i attack you see you at home can't taxes have been at home probably not it's illegal it's not in scope you're going to get in trouble what about renting a botnet with a

zombie that's already inside the machine actually i think that's a really effective way there was a good survey on zeus last year where they said reckon that um they tracked the source ipa addresses of people who were compromised with zeus and they said 77 of the ft of the fortune 100 were represented in the clients that were compromised so i'm a hacker and i want to get into target company a well that's easy i'll just go and get one rent the box it's already got machine inside don't have to go through the perimeter but again pen tester because it's illegal so depending on who your attacker is doesn't work time yeah last one zero days are coming

increasingly prevalent you start to see if it's worth it financially so it's actually viable so if target's efficiently high value someone will take the time to develop a zero day exploit for a given piece of code you can buy it but not legally in a lot of cases so there's ones you can't buy legally so again as a pet tester are you allowed to buy them some of the forms you go on to their organized crime are you is your is it there's nothing in you cool do you think do you think clients can buy off on that i'd like to give three thousand pounds to a russian crime syndicate to compromise your system i just don't think they'd buy it i'd

love to think they would and they should because they're trying to emulate people who will go there but i think that your trip you put that in a sales document you say to a customer i want a budget for buying a zero game oh yeah sure okay i think i'm talking about what you can do in a text i agree legality is interesting sorry clients don't catch any of the known vulnerabilities that would be my argument the the reason i'm putting that up and there's a story i get the next question the reason i'm putting that up um is because i've seen a lot of chat in the pen testing industry about how if you're not using zero days you're

not doing proper pen testing and that worries me because you're dead right people aren't patching the stuff that's out there and if you look at the new verizon thing that came out yesterday the number of breaches that were responsible to be patchable or unpatchable were really low so it's like oh but you've i've seen that argument levied and that worries me because i think people go down this idea of i have to have somebody's gonna go and buy zero day otherwise i'm not getting proper pentax anymore and that yeah yeah and there is another one but within time there is also the fact that some attacks required should have been allocated

you see some companies you see some companies starting to do that like i'm saying and things like social engineering toolkit are starting to push people into the idea that social engineering is an absolutely fundamental part of a penetration test if you are trying to emulate an attacker who would use that technique which is the critical point it's who are you trying to emulate so fixing the problem so the first thing i reckon use different names for different types of job stop buying thing just caught saying i want to buy a pen test and stop selling a thing that's just called a pen test this is just a suggestion there's some stuff on my blog about this as well

actually but maybe you just want a vulnerability scan say you've just you've actually done your threat modeling correctly and you've assessed that the given system is only ever going to be targeted by or by non-targeted people by opportunistic attackers so what you want to do is emulate an opportunistic attacker you don't want to go in there really heavy because what's the point you if you've done your threat world again you've come to the conclusion you only think optimistic attacks are there or maybe you want to use an ultimate opportunistic testing method can any organization outside of maybe government really understand who the threat agents will be that will target isis do you have the intelligence you have

yeah i think you have to take a stab at it because right now the problem is the problem is you can look at someone you say what type of data is typically attacked and breached and card data for example is a red flag you know if you get card data your target you know if you do something which people are going to potentially find objectionable you're a target if none of those things apply are you a target yeah you balance the probability you're right never over 100 percent so wake up tomorrow morning and go i hate the shetland tourism board yeah especially increased activism right well yeah sure but you're gonna say some people are more

likely to get activism than others if you're a security company absolutely you should accept your targeted i mean that's been obviously proven but otherwise yeah if you're doing this correct i agree with you entirely i agree with you threat modeling without threat modeling you're blind you're testing point it's just managing the threat is more important so moving into the shared sports the best challenges if you work for them you're much more less likely to get targeted i would argue that managing the threat right now imagine the vulnerability is all and it needs to be partially saying who am i actually being threatened by and when i'm doing testing i should be targeting that threat model

and you have to get to approximately i agree you're never going to be 100 right but you could be a lot better than just blindly saying i'm just going to do the pen test on my exterior perimeter which unfortunately is where a lot of companies are some companies are better than that but a lot of companies usually just buy the pen test every year because they've been told they have to have the pen test and that that's not the best bag for book and then you can go up from there and and these are all assessments so what i would argue is is that where you have companies who are less mature so they don't understand their security

posture or they don't understand where their vulnerabilities lie but they're far better to be looking at an assessment style methodology so instead of doing goal-based what you're trying to do is say here's what you're weak here's where you're strong but we'll go in probably white blocks we won't go in black box because you waste a lot of time doing black box where in reality they might benefit better be so here's where you're strong here's where you're weak here's where you're doing things badly and we do that more quickly more effectively they're spending a lot of time researching and giving vulnerability when actually you've got password password um so and then test opposite the bot but then once

you've done that yeah so hey actually predicting people how about this threat model needs to be done first um the right type of test for a customer so assess myself yeah so if you've got people who are less developed they should get an assessment style testing if you've got people who are more developed they should be getting absolutely at that point penetration testing style testing goal based testing makes sense but it's actually getting the right type of testing to the right customer

approach to scoping i think the problem you see with scoping is is this idea of one size fits all and one type of testing fits all and unfortunately things like pci and things like the general the use of the word pen testing this is why this pen testing must start it's not that i think that's trying to assess your security is a bad idea it's that i really hate the fact that everything's a pen it's for test health ask for something yeah it's what you call it doesn't really matter the problem is that people you need to start off by getting regulators in the industry obviously officer can use the word pen testing and unfortunately the thing that really

concerns me is it's taken five years to really get the word pen testing in you know you go back five years and people weren't you know it wasn't even assessed after now it's taken five years to get business right to them and you have to do something and unfortunately what they've hooked on to is the idea the thing you have to do is the end text and now you're gonna take another period of time to say okay we've made it that far actually we didn't really mean that what we meant was you want to do some direct modeling you want to assess your risk appropriately and then you want to target your testing so more underrated

has ever heard of the umbrella okay memorize the um they assess things and notably the excess sinks and you know what they don't say about sex when they assess a safe they don't see either yes it was safe or no it wasn't what they do is they say how long is that safe able to resist what type of attack on part of the safe so they say they'll rate it as saying 30 minutes with handheld tools on all six sites that kind of assessment seems to be more like what you should be doing on security and when you're assessing your security what aspect will my industry for how long using walk tools and providing a report that says that

because then at least someone can say here's what i think i've already tested against and here's what i haven't that's what car security devices are sort of so secure it means that an average seat for the power drill will take more than five minutes they're not saying yes exactly but that's my point is is the fact that the moment you look at uh to an extent i mean i'm not generalizing by necessity but there are people in a lot of cases the pen test has been sold as this system has now had been done having the pen test and it is tick box or not tech box and then what you really want to say is

we spend this long with this number of tools on these on this scope so for example explicit saying whether you did or didn't do social engineering saying you kind of you kind of still need to do this stuff because we didn't and social engineering currently this this year his favorite flavor of the year is the way people are getting it and yet he doesn't get tested people might think well i've had the pen test i'm okay no you're not so the underrated labs approach seems to be a better way of doing specifying what does in school resistant for specific duration and it specifies what type of attacker you're actually emulating so those are the slides any more

questions

that i mainly work with for isps and they'll ask for a penetration test and i just say no because that's probably not what i mean because you know that basically means finding one of your customers and then you know trying to break your customer which isn't good customer relations or you know like me he said an iphone app then like your customer their coffee shop their isp apple and a bunch of other people who have far too expensive lawyers for you to uh yeah you even worry about this question give up and so forth

the legal problem tends to be a really nasty one i mean definitely in this world of outsourcing you get companies like amazon and google who take a frankly realistic approach to this where they'll actually pay you to find stuff in their systems and you've got a lot of companies where if you try and touch their systems you can forget it you want to do a pen test on yourself now this aspect of any um type of that type of attack is automatically malicious and therefore illegal but the problem is from a liability perspective if you go testing the reason that they do that is they've got availability out time so the reason that the waste

outsourcing seller stuff the client is it's an sla the sla is uptight this is the security it's uptime so if yes if you knock the system over that's what they risked it then they will lose money on their sla and that's all they care about so if you if it's test not so their incentive is to say no i'm not going to let you test because you might not get over and if you do not get over it's going to cost me money it's part of the thing that we've struggled so hard to get modular testing i've got a project i'm developing this we really want to do some security testing but what we want to do is like test that

bit that you built you secure right or you know it's not vulnerable to the range of attacks you don't want to test the whole environment the infrastructure the people that are going to be using it the processes around it the problem is when it slots in and it slots into an existing environment so it's what's interesting which involves these people and what the business ultimately talked before the business and what businesses care about business care about money if you slot a nice secure thing into an insecure process and then that gets hacked even though it wasn't that you know who's going to take it in the neck it's the guy who said that was secure

even though it was actually something you know so the problem is scope against and you work next

10

it comes down to it comes down to customer education in a large part which is one reason i'm doing this all it comes down to customer education because until customers know that they shouldn't ask for that until customers know that actually you need to actually give it more thought to that and if you ask for that you won't get what you want then they're not going to and until no one asks for i'll tell you what no sales guy will sell it or sales guys might get a lot of those guys they'll just go i can sell this and make my bonus this month or i can spend a lot of time trying to educate the customer and i might or may

not make a sale so i'm going to do it easy way

you were mentioning the um don't know what they want to buy you were mentioning earlier on that you were comparing the security models in those areas you know what the threats are you know what the attack factors are you you can identify them quickly yeah and they are well known i'm very likely to be a pc right now i've known that in your world there are days yeah it's something you can it's one of the problems is every day what the problem is testing five years ago isn't good testing now what the the good attacks were five years ago are not the same now absolutely um that's kind of what makes it a challenge because it won't stay put but you know

and that's also why and that's the next place where at the same point we were talking about different uh i mean scope testing where when you have a testing company you get to test one specific module or one new functionality an application and then you get someone else who puts it together and the system ends up being vulnerable because it has never been tested once assembled and it's always the investigation yeah or you buy it you buy it you buy a component from someone else and they didn't do their job yeah in terms of the process the business doesn't understand that and the reason why you didn't understand that and that's how you close the loop

because you don't understand security yeah yeah they will get mistakes so yeah you're right it's it's all about maturity but can you start addressing that well i guess you work with a customer today as a fantasy company and you try to teach them and i'd argue you won't get funded to teach them what you mean by security and to teach them how wrong they're gonna yeah you should pay your money on yeah on you teaching them so i'd obviously industry i'd argue that for example the uk crest is a great example crest should be because they that's exactly what they're there to do they're there to say we're an industry body all pen testing companies and we should

be looking at some budget hopefully from either customers or suppliers to say hey guys here's how you should do this properly um here's some practices you can use which will improve that so crest and or the government and or regulators pci being a bit more granular

and the problem is that the threat moves very very quickly so the i mean it's a classic problem you're hoping there's corporates who move very slowly by their nature you know fifty thousand percent companies not move quickly um attackers will innovate next week because they found the new vet or their new way of doing it and they'll just go and then verizon verizon for it came a couple of days ago it was fascinating because it showed how many basic completely basic attacks because they'll just go in the easiest point they won't go off you know so they were saying that the best thing to do is actually just focus on getting a low level across everything

because frankly then you'll be up above most of the opportunistic attackers so unless you're a real target of choice focusing on getting the basics right everywhere is actually more important than focusing on having really high stuff up here but in one place because they'll just come in the side um and i am done fantastic one more time there's a point earlier about how do you educate customers and make sure that they know they want i think that if as an industry we can't give the customers what they want without them knowing what fears they're doing to the top because you know like good user experience is about giving people what they need

the problem there is that sales incentives what the salesman's incentive it might be different from what's good practice which is fortune of the dream i think i am over time

you