Do We Still Need Pen Testing?

fast forward we can stretching okay the agenda today a little bit about me which I've been talking about what I'm thinking of why the talk why why are we talking about pen testing basically i'm going to give you my thoughts and ideas and hopefully was some time to discuss it at the end of the time there's my contact information if you guys ever want to get in touch with me by the end of the talk if you still want to speak to me that's what I used to look like back in my pen testing at NSA days when I had hair and had power you guys have ever heard of Paul security weekly I'm one of the hosts

there I try to get on there least once a month I'm getting on there more often now since i'm actually currently unemployed i'm also a Jedi Master there's a story behind that which I don't have time to tell but really I'm a Trekkie the original series not any of the cheap imitation remains my 33 years of information and security experience starting at in and say ending up as aqsa and then you know the last couple years just been doing kind of the evangelism curmudgeon role to put things in context when I started it and I say everybody know what that thing is what is that thing this is interactive things it's the Enigma thank you it was

still a secret that we we and our allies the Brits had broken that why do you think that is nope there are countries still using it it was not revealed i started in 86 at NSA it wasn't revealed that that technology had been broken I want to say to like 88 89 something like that while I was it and I say primarily as i was telling you get ripped analysis i bounced around i was actually there there was a little conflict in the desert before the last 13 years called Desert Shield Desert Storm I was involved in that from the civilian side of things on the on the operational side and then what I was telling you guys

initially was i got into penn testing at NSA we were setting up doing ethical hacking pen testing because the office that i was in was fielded systems evaluations i gave a talk about my old days as a creepy can find it on youtube it's called Tales from the Crypt analyst I ended up in this office it was called fielded systems evaluations NSA discovered or somebody had this brilliant idea that the way that we often break the adversaries communication systems and encryption systems is we as people don't use them right they reuse key they forget to switch some switches they find bat bypasses is if you know any of this sound familiar I disagreed with one

thing the keynote said that he said something about how pen testing has changed so much in the industry has changed so much over the years I'm like yeah I don't think it's changed much at all that's another talk but because we were doing fielded systems evaluations and there was and networking was becoming more popular we started looking at network systems which got us into pen testing and ethical hacking and vulnerability discovery and so on and so forth anybody heard of this book now that you're going to get a voucher for a free book I might recommend this it came out earlier this year it's a book called dark territory the secret history of cyber war

there's a chapter chapter four called eligible receiver and in that chapter there's this following statement the NSA had a similar group called the red team as part of the information insurance director director had formerly called the information security Directorate the defensive side of the NSA stationed in Phoenix the building out near friendship airport during its most sensitive drills the red team worked out of a chamber called the pit which was so secret that few people that NSA knew it existed and even they couldn't enter without first passing through combination lot two combination locks doors are you intrigued or you excited the pit is the name of the office I used to work in so we're famous this is the fanx

complex they actually spelled it wrong in the book is just FNX this is right outside of bwi airport if he's ever flown into or out of bwi airport and it's a was most of those buildings and still might be today for all I know and friendship annex BWI Airport used to be called friendship airport so it was just friendship the airport annex friendship and it the pit was in that building there and it was on cash flow floor his own it was 20 years ago I want to say we were on the second floor in that corner I give another talk about those times be on the lookout for it actually submitted it for shmoocon hope so waiting to see

whether they'll accept it or not but again that's another story for another day pentesting I started to tell you guys I came out in the commercial world I did pen testing for about five years in the commercial world and then got away from it got into doing PCI work USA work and you know sort of wasn't doing anything technical per se for quite a few years and only got back into it the last couple years when I started coming out to conferences again so I'm at Derby con 2014 ed scotus as the presenter he's giving a keynote on how to do the how to give the best pen test of your life and I'm like oh

that's cool i'm going to learn about how pen testing has changed over the last 10 or so years since i've been sort of a way from the hacker community and so I'm sitting there listening very anxiously and and these are some of the points that he had you know this talks up on YouTube you can find it these were his key points the best pen tester of your life has great value provides previously unknown insights so on and so forth he talked about know you've got limited budget living a limited amounts unlimited amounts of time you can you know go everywhere do everything get all the best results but it kind of left me I got a I got a

little bit irritated while I'm listening to this talk it bugged me a little bit because first of all he didn't define what a pen test was and I think that's kind of important because if you don't know what you're getting into how do you know if you're been successful or not even even beyond defining a pen test just defining what the goals of a pen test are I mean what is a pen test anybody what I mean assuming there's no right or wrong answer anybody want to take a stab at what is a pen test test controlled assessment of the network okay I mean these were you know that's keep going I mean a pen test can be many many things

I found in the early days when I came out it's been 20 years ago into the commercial sector everybody wanted to have a pen test because everybody was plugging their networks and their their what used to be closed internal systems they were attaching them to the internet and they wanted to know how to secure it and of course the industry was born where you know first he had to go out and do assessments and scanning tools became available firewalls were available to protect the perimeter and I'm sorry yeah I mean all sorts of stuff was going on but it all started with it all started then with you've got things in place already let's do a pen test but

really when I would talk to my customers and ask them what do you want to get out of the pen test and then I don't know I just know we need a pen test and I'd say well do you want us to find a way into your network or do you want us to find all the ways we can get into your network invariably they said well we want you to find all the holes we want you to plug everything you know discover everything I said okay but that's not a pen test that's a vulnerability assessment so really pen testing what I did prime early in the early days was far more often a vulnerability

assessment there were times where there was exceptions but most people wanted to know everything that was wrong that's a vulnerability assessment I go away for 10 years I come back we're still talking about vulnerability assessment so you know again there's no necessarily right or wrong answers it's what do you want to get out of the pen test you know what's the goal of the pen test back in my day when I was doing it many years ago the goal was to get root nowadays it's more administrative access but I was doing mostly unix based networks back then so you know we would we had our little fun stuff we did you know we got the root dance and we had

all sorts of little stupid things we did high-fiving each other when we got root sometimes the the objective was can you get here can you get to a particular system can you get to a particular data set or database we had an executive at NSA put a file in one of his directories in one of his folders and the objective was can you get to this file and of course we got to it and we added a message to in response to the message that he'd written in this little text file you know is it discovery is it I used to hear this a lot talking to customers we don't do that we don't store that data you know are you

capturing sensitive like social security information or in the credit card industry are you capturing and storing the sensitive authentication data the three or four digits oh no no we don't capture that well you're accepting it here on your website yeah yeah but we don't store it well inevitably when when customers said we don't do that our job was to discover oh yeah you do and yes it was an educational exercise perhaps lots of different reasons and they're all legitimate really if you just go into the pen test with this is what we want to get out of the pen test one of the things I've also seen in the gap over the years is sort of anybody

know what that is reference Kobayashi Maru it's a movie reference from Star Trek which one come on shout it out we don't have time its star trek to the original series the Wrath of Khan the the spoiler alert if you've not seen the movie one of the themes of going through the movie is Captain Kirk is the only cadet at Starfleet Academy that ever defeated this unbeatable test which was called the Kobayashi Maru the point of the test was to see how a cadet would react in an unwinnable situation so it's it's sort of a holodeck you know roleplay there on the bridge and they're being overrun by Klingons and the ship's about to be destroyed all lives are

going to be lost how do you handle that that's the point of the exercise Captain Kirk was the only one who'd ever defeated it and it turns out at the end of the movie what he did was he hacked into the application and changed it so that there was an outcome where he could win boy you know that's what you know pentesting in the early days you know people got good at the firewalls and the perimeter protections and we started talking about but what if you're an insider where can you go from here social engineering came about well you know what if people are breaking in what are people are sending you phishing messages we've learned a lot about that

this morning so anytime a defense goes up you figure out a new way to you know side sidestep and get in so is that legitimate or is that not legitimate these are the questions that I asked so I mentioned I was aqsa I I came into this talk a few years ago and my mind mindset is really based on the PCI standard if any of you guys ever seen this before is this new teeth the pci data security standard has six overarching goals which i would argue or pretty comprehensive technology independent and they're split up into 12 major requirements and from there it's I've lost count there's four hundred and some-odd sub requirements and I did say PCI so go ahead and

insert your your stupid PCI auditor now these are some of the PCI cut actually these are most of the PCI customers I had over the years in the middle there TJX companies heartland there's other companies up there they claim their claim to fame is that they were famously breached in the mid mid to late two thousand so the team that I was involved with was responsible for going into a lot of these breached company companies in the in the mid to late 2000s and and do the cleanup and get them really on board with PCI and security so not the typical I'm not the typical pci qsa the data security standard in version 3 which came out

into 2013 they finally introduced detail into what a pen test is supposed to be PCI is always required pen testing since the very beginnings of the standard 12 years ago but they never really defined what it was I looked at a lot of pen test results as my in my role as aqsa over the years you have to do it inside you have to do it outside but there was really never any definition of what it was supposed to be any definition of should it be automated or manual or some combination you know what is it exactly so finally pci came out with a document that kind of put it in detail and then they put it into the

standard this is what's supposed to be in a pen test and it starts with have a documented methodology what a great idea have a set of ground rules and and you can read through that it's it's pretty comprehensive inside outside what does that mean and oh not just inside outside your network but inside outside what's known as the card data environment you're more sensitive systems do it at the network level do it at the application layer great idea they have this nice definition of the pen test and we don't have time to let you read it all let me paraphrase it's pretty good and it says and this is something that PCI is famous for they get down to

really what constitutes a pen test is exploitation or attempted exploitation of discovered vulnerabilities risks holes bugs whatever you want to call it where they failed in my opinion is they they say it should be a manual process but they didn't say it must be a manual process the pen testers that I used to work with we kind of prided ourselves on it being a manual process and I think a lot of the pen testing community a lot of the eds curtis's and the big names they all talk about it really should be manual it's got to be you against the world not where they call puppy mill automated tools and things like that but they

allowed it because there's a lot of companies in the PCI world that are selling automated pen test tools that companies use to to meet the requirement so the operative thing here is exploitation the gap is between manual automated that's my paraphrase but here's here's what really got to me in the world of PCI where companies are supposed to be basically implementing a comprehensive security program and they're all supposed they're supposed to be it's sort of linear building secure systems building secure networks maintaining and keeping them secure having endpoint protection in place having monitoring in place having logging in place having intrusion detection in place having a vulnerability management program in place it's patching systems on a regular

basis as critical patches are coming out you know responding to events revisiting everything as new things are discovered doing vulnerability scanning you know the scanning tools that are out there are you know they try to stay up to date with all the latest greatest bugs and vulnerabilities and miss configurations that are out there and if you're supposed to be doing all of that do why do you need a pen test any initial response to that so let that question simmer for a minute the things that I saw most often especially in my PC I days of why people wanted to do the pen test first and foremost was to meet the compliance requirement again most companies wanted to treat the

pen test as a vulnerability assessment find all the problems we did in our days of pen testing when I used to do it we started dabbling into the testing of the the defense mechanisms especially when IDs was first coming out you know it became kind of a cat and mouse key you know if you do a certain amount of activity do the detection mechanisms trigger an alert and you're caught and so we used to come in stealthy and gradually sort of turn up the volume unfortunately back in those days and I don't know if it's changed a whole lot we'd kind of you would kind of spin the volume to 11 and still nothing would happen they still

didn't detectives and what we tried to do with our pen testing results was tried to make it turn the results the reports into learning opportunities hey here's things that are wrong with your network hears things with the fundamental ways that you're going around doing networking or just simply you're unaware of what you've plugged into and what are the features and the capabilities of the technologies that you're using

in my opinion and this is where I'm going to be a little soapbox e I kind of felt like you know pen testing has been around for like 25 years give or take some could argue longer but you know you know as long as network systems have been around there's been some concept of penetration testing and what was nagging at me is again if if people are doing all the right things from a security perspective from the building a program perspective why still pen test especially pen tests and I I think I've seen it enough in the last couple years that I've been coming to these hacker conferences listening to a lot of talks about pen testing there's

still a lot of pen testing that's done in my opinion for the purposes of vulnerability assessment like this whole thing called security that we do starts with a pen test starts with discovery and while I think discovery is legitimate and pen testing is fun I wonder about the especially if I'm doing it as a consultant or a third party for a customer you know is it really the most cost-effective way to educate and aware and find problems and find find issues and goals and misconfigurations and things like that so where my head is is that you know penetration testing in terms of what it is and what you're doing in terms of the goals should start to be more focused on

not vulnerabilities because you've supposedly got mechanisms in place to find and discover and keep keep your arms around vulnerabilities and by the way they're pervasive and ongoing and they're always going to be more more vulnerabilities but maybe pentesting should really start to focus more on the threats it was talked about a little bit you know a specific type of threat or threat agent was discussed in the keynote this morning but yeah how do you protect against phishing if that's a common way for the bad guys and the attackers to compromise networks there's more automated attacks that are out there today when when I first started out it was people trying to break into systems and

you actually breaking in getting through the firewall getting into the internal network and what the hell is an internal network these days but the you know so the technology's changed the landscape has changed the fundamentals the the philosophies I don't think have changed a whole lot there's bad guys and I think I left his talk early I don't know how much he went into this but there's bad guys there's threat agents that may or may not want to do something to you your company who you're working for your customer it's good to take time to figure out who that might be and what their motivation might be take a step outside of the technical realm of everything he was touching on

that a little bit but here's where I was at my idea of the ideal pentest yeah you have a good solid methodology you know what your goals are all parties agree this is the objective of the pen test but and again with my thinking that this is a customer this is somebody that I'm trying to help become more secure and to protect whatever it is they're trying to protect at the end of the day when the pen test is performed I get no results it's a fail that to me is the ideal pen test because the company is secure or my attempts the pen testing attempts are detected and detected early and blocked or at least you know the red flags go

off the alarms go off and there's a response that to me is a perfect 10 till pentas not one that finds more vulnerabilities and if you ask a lot of pen testers you know security weekly a month or two ago we had a guy on that it that works for pen test company and he had gone through a bunch of their customers and just did a survey of how did we break in how did we get access having access global domain access and he categorized it in the top five ways was exploitation of trust relationships passwords password a week shared passwords common passwords some variation of passwords and basically lack of segmentation a flat network you know everything can see

everything I'm like that's what the f it was 25 years ago why are we still doing pen test finding that stuff when I used to do pen tests if we found something like a default faster password we'd stop the exercise and say we don't need to be pen testing you we need to we need to spend your money educating I had a customer one time that they had they had just installed a firewall as a brand new firewall top-of-the-line firewall 20 some years ago he probably never heard of the product they've gone through all the training classes they got their servant certificates they set it up this is back in the days when firewall rule

sets were like 25-30 rules so it's pretty basic the idea of the DMZ was actually you had actual Network segments that were sort of linear not multihomed like they are today not all in a virtual device like they are today well we started poking him from the outside and was like there's no firewall there and we went inside and we would start on the outside go on the inside and while we were on the inside they started getting dose of course we were the ones blamed so we said timeout let's look at your firewall rules well we're looking at their very short list of firewall rules and the final rule was any any allow so I said timeout we're done we're

not doing a pen test anymore we're gonna we're going to teach you guys how to architecture network and set up firewall rules that actually work and and that was a seminal moment for me in my career because when we presented this to the customer I want to say it was their CFO he's like wow that's the smartest engagement I've ever seen because usually when we hire people to do a job they just do the job and take the money and you know what you paid for us to do it so that's sort of that that helped to mold my career in my attitude that I should always help the customer do the right thing regardless of what the

contracts is so anyway I really think we're not there yet as an industry pentesting still comes up way too much in terms of vulnerability discovery and while I think pen testing is important and legitimate I think it's a safety net exercise it's a live-fire exercise that you do after you've done all the things that you think you can do to protect your network they're really it's much more threat based it's it's being creative you know what if we do have an insider what if we've pissed off somebody in development or what if we pissed off pissed off somebody in accounting what if we hire third party cleaning crews are coming over night you know it's okay and it's legitimate to

test all these different scenarios hopefully because you've already gone through the exercise of what are the bad guys who are the bad guys why would they come after us what would they be trying to get what are we trying to protect at the end of the day so just applying a little bit more logic perhaps and more of a flow to it not just simply well pen testing as a way to discover more vulnerabilities a lot of the guys on the security weekly show or pen testers and they talk about oh I just did this really cool job and we found a default password and like how are you writing that up as a vulnerability

because to me that the finding is not i found a default password to me the finding is you've got something wrong with your security program that allows the default password to be out there in the first place there's something in terms of the processes and by processes i primarily mean the manual the people applying and doing all the things one time we did a pen test where we were hired to just look at the firewall single interface single IP address we were hired to just can you break in through the firewall and while we were just doing sort of scanning of their environment we saw web servers and they were not part of the scope of our engagement but

we poked enough to say these look like default insults this is back in probably windows in T days and out of the box systems internet-facing probably we're talking inside as well as outside and we didn't touch him we didn't go beyond the contract of the bank or an insurance company that would have sued us if we went beyond but when we reported our findings we said you know your firewall is great but there's a stuff over here that we're pretty sure we could have used to get in and and break into your network another seminal moment in my career the whoever we were presenting to at that point turned to his people and said what's wrong with

our processes that were allowing our people to put these boxes out on the internet and I was like wow that's exactly the right question to ask what's wrong with our processes you mentioned somebody mentioned risk earlier we don't teach the risk equation anymore everything in this industry is focused on vulnerabilities but guess what vulnerability is just a portion of an equation that evaluates risk and there's probably 10,000 mathematical equations out there to attempt to try to put a number of value to risk to me it's got to be higher level and more intuitive and I've always tried to break it down to simple math you know there's and this is what I learned in my DoD days you have

vulnerabilities you have threats there's the things it's a Jew to protect against the vulnerabilities and the threats that is somehow reducing whatever this value is but there's this overarching multiplier especially in terms of the commercial world not so much in terms of the DoD of the value of what it is you're trying to protect what it is you're trying to protect might be sensitive data it might be financial data it might be consumer data it might be your rep you corporate reputation how do you put a dollar value on that how do you define but define what the risk is and how much you invest against protecting it against the bad things happening I'm sorry

especially if it's qualitative and you know I would talk to many customers over the years especially in the early days you know what do you not want to happen well we don't want to get our names and the used to be in the newspaper now would be on Krebs right you know we don't want to be the next company that's you know victimized and has our name plastered all over the place them okay how much do you want to protect against it I mean the story that I that I will tell hopefully as shmoocon is about one of the early forensics exercises that we did for a customer that had to do with just their website

being defaced I mean who cares about their website being two-faced well large companies especially in the early days it cared about corporate reputation and what what how might this impact our value our stock value our stock prices our corporate image our customer and you know customer confidence and conducting business with us hard to do we need to talk more about this and oh by the way I've heard people say this just in the last couple months because I've gone out to conferences and trade shows ins and I've seen companies that advertise we're experts on threat I'm like what's a threat no what's a risk and like no it's not it's part of an equation that leads

you to what risk is so we need to learn what these terms are and what they mean and you don't have to agree on an exact absolute value but for the love of God have to understand have some understanding of the interplay and how these are component parts of this overall nebulous thing that we call risk frankly we're in the brisk business we're not in the security business I mean that's my bottom line you know how do you do this and this sort of bleeds into some of the other talks that I that I've given that are out there you know we need to become educators we're the ones that know what's bad and know what's wrong so we

need to figure out how to convey and communicate that to people I've heard referred to as muggles or the stupid people or the real people or the ordinary people which implies we need to be humble and stop calling the people that we're trying to convince to do things differently implying that they're stupid I can go on and on about this but as a consultant to me it's all about communication and it's really perfect persuasive speech you're trying to convince somebody to change behavior spend money do something differently invest in different things that where most of the commercial world just wants to buy technology drop in some new product and it's going to do everything for me so I don't have to think I mean

that's really the that's really what we're up against a lot of times we've only got about five minutes any questions comments pushback feedback agree with me disagree with me it's okay yes sir

mm-hmm yes so I think this goes back to what is a pen test you know a lot of the research community vulnerability research community bug bounty security research community there's all legitimate reasons for trying to find vulnerabilities and bugs and things that are wrong with the technologies that we use I just don't think that's a pen test we call it a pen test we kind of lump it all together but to meet security research and vulnerability discovery is a different exercise than pen testing that makes sense yes what's your thoughts on the execution standard I haven't read it you guys I mean the company's I start accepting it or I mean that we built our

company methodology I have that referenced in there somewhere but I haven't looked at it in a while what I think isn't actually I think it's referenced in the in the PCI documentation or or maybe embedded Bush will be what's the you know truly the benefit to a methodology in a pen test and the importance of that versus so yeah I guess depending on what methodology you're following or what industry accepted methodology you're following I think the importance of following a methodology is understanding what the goals and what the outcomes are I mean I used to ask my customers okay so we're going to do this exercise whatever it is we're going to produce a report what are you hoping to get out of

it you know what it you know who's the consumer what are they looking for especially in the PCI world I you know you might imagine as a qsa I sometimes went into hostile environments because I was just a stupid auditor I happen to know a little bit about security so sometimes I had to kind of win when the technical people over that I had to talk to and one of the ways i did that is look you guys understand your network far better that I do and I'm sure you're you know what you need I'm your best friend right now because if we can slap pci on it guess what you're going to get it and

lo and behold they would and i won many friends and again that's part of the trust relationship yeah all right we got to wrap it up sorry guys that we got impressed thank you