Why Pen Testing is the Last Thing You Should Do by James Bore

yeah why pen testing is the last thing you should do or alternatively if you like subtitles how to use a superficial controversial sounding title as clickbait first thing and hopefully this will work because this is experimental in the first time I've tried something like this in a presentation if you trust me you can try the QR code there otherwise if you go to that link there should be a nice interactive thing there which will let you do any questions you might have so you can throw them in and I may or may not look at them and answer them at the end but you can just put them down as we go but more importantly it will

let you answer the pub quiz polls so I'll give you a moment to do that I'm not sure if anyone can hear me this is a very one-way thing sorry you moved to the microphone as well so you can hear us yeah brilliant brilliant okay if I can keep hearing you that would be great yeah so I think that's enough time for people to have jumped on to the QR code if they're going to like I said I hope this works so first thing College if my computer doesn't lock up there we go on the floor right a pub quiz question number one what is pen testing so you've got writing a lot of reports

to test Barrows you have using attacker-like techniques and tools to breach Target system or running nessus and filing the logos off the report

okay so with one answer we've got 100 on attacker-like techniques and tools

three answers ah that will do that's representative of the audience yeah so everyone should know what pen testing is it's you acting like an attacker try and breach a Target system it does also involve writing a lot of reports but that's usually on computer so you're not testing Burrows there's a lot of pen testing out there which is running nessus or running a vulnerability scanner and filing the logos off that is distressingly common and if you've got any experience in security and get one of those reports it's immediately obvious and that's when you add that company to your list of never work with again true next question what's the actual aim of pen testing what's the goal of it

are we looking to meet compliance requirements are we looking to satisfy customer demands find security flaws in the system improve the security of the system which of these is ultimately the main aim of penetration testing

people

not sure if that one's working so I'll carry on for now and it's a vulnerability scan bad pen testing yes no or it's more complicated than that now this one I want to go into for quite a bit because yes a vulnerability scan can be counted as lazy pen testing it can be counted as trying to rip a customer off because you're not actually doing the work they're paying for but if a company if an organization hasn't got any vulnerability Management in place then there's no point doing a pen test you will just be wasting your time because you're going to find less that way you're actually doing better for them by running a vulnerability scan

they'd be better to do that themselves if they want to give you 1500 a day to do it for them then okay I can see why you might go down that route foreign and finally for the questions at least when should you pen test so lots of the time you'll get annually sometimes you might get monthly if someone's particularly ambitious after significant changes though I've never found anywhere that can tell me what a significant change is or when you need to which again is really hard to Define and yeah annually is the most common whether that's right or not depends on what you're trying to do so moving on from the experimental public so we're going on the journey uh short

Journey this one of development and penetration testing and unicorns and clouds and magic and this is one that I have seen in more than enough organization I would say the majority of organizations I've ever seen or worked with have a very similar journey to this the approach we're using for development doesn't matter whether it's devsecops and we're taking that approach whether we're using agile whether we're building scrum whether we're using extreme programming which I didn't know existed but apparently that's an actual formal approach to programming I don't know what it involves I don't want to whether we're using spiral which makes me think of spirals of Doom which is probably quite accurate lean Ward full whatever

you might want to use that doesn't matter for this journal because we're going nice and Broad and generic and what we do is we start out with an idea brilliant we've got an idea for a project we write it down we take it to a pitch meeting we pitch it we get budget we get resource whatever it might be however that works internally for the organization that's what we start with just an idea then we design and build it we don't care if it's Antioch and we do that iteratively over two weeks of iterations of stress and anguish and coffee or waterfall and we do it over 18 months of stress and anguish and coffee

how often we do it we design it then we build it and then we pen test it to make sure it's secure let's give the pen testers five days to be generous it seems to be about an average amount for a pen test they run a vulnerability Scan they run their automated tooling if they have time they might do some manual exploration good okay they write the report that's at least a day or two of their time but going through the findings from that vulnerability scan and that automated tooling is going to eat up most of the time they would otherwise use for manual exploration so how much value has pen testing added here if they find vulnerabilities with

the scan with the automated tools manual exploration or they find nothing and the answer is odd because really they are adding the most value at this point by doing the scan and using the automated tools because that's going to discover the most flaws in our software that's going to find the most things we can fix and cause the biggest Improvement to security which is a complete waste of a pen testing budget and a pen testing company and of pen testing skills so what's happening here is you are pen testing too early it's the first thing you're doing for security not accusing any view of doing that but there's plenty of companies who do if pen testing is the first and only thing

you are doing for security you are wasting huge amounts of resources so let's go with a slightly different approach this one is a magical journey of self-discovery security tools imagination unicorns and software development and I've seen this actually achieved fully I would say about three times in 20 years but when it has been it's incredibly effective so this time we start with an idea but we do a bit more than that we Define the security requirements because we can do that with just an idea yes they'll be high level that we can still Define them we do a bit of high level threat modeling we consider what the threats might be we create our threat profiles

we stick them up on the wall for the rest of the project we run a quick risk assessment looking at the basic functionality that we'll need for this idea to work and what risks that raises and throughout all of this and beforehand and after and all the time we train our project teams on security appropriate to their role so project managers get an idea of higher level security developers will be focused more on technical security and secure coding but everyone is constantly going through that awareness and that training then we design it and we do more thorough threat modeling so we start applying whatever our threat modeling methodology might be whether it's stride whether it's pasta whether

it's octave whether it's something else that we've built entirely in-house which is the right way to do threat modeling and we select third party components from our catalog where we know that they are reasonably secure and we know we've got the infrastructure in place to maintain them and we've got the intelligence in place to pick up any alerts about security that come out for them and we Define the security metrics we'll be using for the project because we're designing it we can Define them at this point we know the sort of measures that we want to use and we decided on the approved tools are we going to use Visual Studio are we going to use notepad doesn't matter that

it will be one set of tools that we are using that every developer is using to build this they will be approved they will be secure they will be able to do what the developers want there's an important caveat here that if there is no suitable approved tool you find one and you add it to the list and that needs to be fast and easy and then we build it and as we're building it we're constantly running static code analysis and we're constantly running Dynamic analysis every time it's built because we've got security requirements defined early and because we've been using threat modeling rather than just saying old security is non-functional you can't test for it instead we have

functional unit tests for security those will be misuse cases or failure cases and we're testing to see whether fuzzing will overload an input or whether it's possible to inject into something or whether it's possible to elevate privileges but we're testing for failure and we're making security a functional test we've got vulnerability Management in place we're doing scanning we're looking at all of those third-party components and constantly making sure they're updated and we're still doing security training regular reviews throughout the project and then we pen test to make sure it's secure but the difference here is we give the pen testers all of our vulnerability reports every single one or certainly the latest ones and our system test results

and all of the documentation that we have of the system and we tell them we know these vulnerabilities exist we should have resolved them but we know they exist at the moment and we're working on them so we don't want you finding those and we don't want you finding anything that our assassin desk picked up because you know it's there sure go do proof of concept on it just to confirm it for us but we don't want you exploring for them what we want you to do is what's valuable what you're good at try to find the holes in the system and at this point if they find vulnerabilities with the scan well that's worthless they've just wasted

their time and asked because we gave them the scan and if they find it with automated tools again largely worthless we've been running those tools ourselves if they find film abilities and attack pulse with manual exploration and can build up attacker scenarios that's incredibly useful we now know what's plausible we now know what's good to solve or bad if we don't solve and if they find nothing we can be more confident than we could have been in the first one if they find nothing because we know we've done a lot of work around the security so this all ties into an idea of keystones and capstones which is something to do with architecture or civil engineering whatever you want to

call it there's a difference between these two approaches so a keystone is something in an arch usually the top Stone of the arch is the last Stone placed and then you remove the bracing and the whole thing falls apart without that Keystone if we move the Keystone from a bridge it collapses if you remove it from an arch it collapses and what's happening a lot of the time is that pen testing is being used as a keystone which means if it fails if it's weak your security is non-existent whereas what we should be doing is looking at the Capstone option where we're placing it to signify that we've completed it not entirely decorative cap stones are

designed to protect walls and softer materials below but it's not a supporting component it's not critical if you remove it the whole thing doesn't crumble it becomes more vulnerable but it doesn't collapse now in theory this should let you ask questions or I'm sure you can shout from the audience I don't know how they're handling things there today that uh any questions or what questions do you have other than why use my battery on my laptop

it would appear we do not have any questions okay right well in that case I will leave you with the obligatory self-referential LinkedIn self Twitter quote post thing saying uh pen testing should be a Capstone of security not the Keystone if you look for it by the way this post doesn't exist I'm just very good at forging these things I couldn't bring myself to do an actual one just for this um and that's it okay thank you very much thanks