Pen Testing Your Incident Response - Ryan Linn

so contributed to some open-source projects as well what we're really here to do is to let's try that again the monitor just blipped let's try this again for we're gonna be talking about pen tossing and specifically pen testing your instant response program one of the the challenges with pen testing is it means lots of things to lots of different people and I think for the purposes of today what we're gonna be talking about is pen testing as a manual process a lot of the people are saying okay so pen testing is a vulnerability scan with something else and so one of the challenges I think with with pen testing is when you're you're doing with

skilled people and you're dealing with a highly manual process you should really never hear things like well I'm waiting on the scan to get started so I can start pen testing and so if if you ever do the really watching people do that process it's kind of indistinguishable from navel-gazing so you probably want to find somebody different to be doing your pen testing one of the the biggest challenges for what we do for pen testing in general let alone pen testing your instant response program is the preconceived notions that people have there's a lot of reasons that people don't want to get a pen test and what's really interesting is the market it's kind of changing for for pen test right

now a lot of people are sort of going with a more commoditized service instead of very highly customized testing for individual locations and so some of the the other challenges are people sort of assume the worst and so they don't necessarily get what they really need because they just assume that well we're gonna fail this thing anyway and then the other one is if you're actually buying pen tests now it's hard to sort through the industry jargon about what you're actually good and so that's I think probably one of the biggest challenges right now so many people mean so many different things when they say pen testing it's it's hard and then the other one is when you

actually start talking about things like pen testing a security program and incident response and I'm just dealing with the Red Team Blue Team side there's some real disconnects there and what we're gonna be talking about today is how to bring them together and I think one of the interesting things is fear is is one of the hardest parts about pen testing people are kind of scared to to love the pen testers into their organization some of them may think that it's like a bull in a china shop or just sort of the energy or takes over and these are actual real or quotes by the way but I think they're all appropriate and so not being able to get business

justification I wish I could say yes but I can't and some of the people have gotten the same pen test findings year after year there's just like well and it's just gonna be bad anyway so why bother those sort of things and so people Rd yes sir so that's a great question the the question was if they're seeing the same results year after year why are they bothering getting a pen test and one of the biggest challenges in the pen test industry as the difference between strategic findings and tactical findings one of the things you see a lot is people give you a list of all of the different systems that are broken in a specific way they'll give

those to ops op says we fixed all these next year there's a whole nother set that's broken in that same way so the problem is is they're not giving the information they need to be able to strategically fix these so it's not I'm fixing this box this box this box it's we're fixing this throughout the organization and I think that that's one of the bigger challenges right now as well is a lot of people are being cranked through security programs are very much tactically oriented because they've never done operations once you start doing operations you understand how that works you know okay so this is how stuff needs to be integrated into business processes this is how all of

this needs to come together in order to be able to help actually build up the information security program not fix this box this box in this box so one of the the things only to focus on is sort of the difference between sort of the type of pen testing where people are just getting this tactical and the the strategic ones and so let's focus on some of the reasons that people think about giving pen testing compliance is probably the biggest one one of my favorites is we'll say it validating concerns because really if you have a situation where security department is saying we need to do this and everybody is like no no no no we're

fine we're good so having a third-party consultant come in and and find those same sort of things gives you external validation and so external validations one of the big reasons to get a pen test one of the other things I like to talk about when we start talking about security is anybody have a good definition of security okay so there's a number of different constraints and that sort of thing in order to make sure that you have control over an environment right security is an emotion security is how you feel right now we feel pretty secure dude walks in with a gun that changes pretty rapidly the way that people make decisions on security is

completely emotional a lot of the times things are backed up by numbers but it's how you feel about those numbers from the most part especially when you start looking at top level executives and we can see this a lot in our government every time something big happens new laws come out regarding cyber security they may not necessarily be the most informed but they're an emotional response to something that's going on right now and so there's a lot of emotion to the - security in general and so demonstrating impact when somebody can see something it makes a bigger impact for them to be able to be willing to invest what it takes to get that um selling political wars between

factions is kind of interesting a lot of the times there's a lot of disagreement between security teams and other people as to what needs to happen so that sort of thing and then establishing a baseline for security posture is also important because unless you have some base level of posture a lot of the other stuff we're gonna be talking about next isn't really going to matter if you can't actually see into your your infrastructure and have enough security to be able to to detect problems then you're not going to make it very far as far as an instant response in general so we've covered some of the reasons why people don't want pen tests but let's

focus specifically on pen testing and instant response and so obviously our goal is to win right our goal is to make sure that we are creating the most secure organization that we can and that is both as far as the control side that you mentioned making sure that the things are there to to help you feel secure and for the executives to be able to sleep at night the security guys to sleep at night and you know one of the things that you you you hear people asking a lot more now is what is it that you're worried about not you know where your gaps but what is it that you're worried about and so we want to win and

one of the great things that security the pen testers are good at sort of changing the paradigm and so what we want to do is we want to change how people are looking at this at this problem and really start using pen testing as a multi-tool not just going in and finding individual problems inside an organization but working towards actually making the organization better as a whole and looking at not just pen testing the individual systems but actually pen testing processes procedures and everything else along the way so some of the other areas that I think that this is the ability to help with obviously product certification making sure that as you're rolling out product not necessarily like this has

Ryan stamp of approval but making sure that you're actually following best practices those sort of things so that you can say your product has gone through the amount of rigor that you wanted to before it's released to the public or if you have architecture changes more more organizations or having to rapidly change being able to validate those before their they go to production is great the rep modeling is a big one if you don't know what you have and how its vulnerable how do you know how to protect it and then things like strategic prioritization going back to you know why do people have the same findings every year strategic prioritization is how you determine what

projects need to be higher or lower in that priority list in order to make sure that the right things are happening to have the biggest jumps in your security posture and due diligence is really big right now people I probably know that you can get breach insurance these days right so if you don't follow due diligence that is very little good so you need to be able to prove due diligence in order to be able to make sure that actually matters one of the other ones that has been popping up a lot more is a lot of the the people are changing with in information security rules and so if you land in any position

maybe as CEO of a new organization how do you know where you really stand as far as your organization pentesting can come on and help you identify what your current posture is so that you know what you need to be working on especially if it's something that hasn't been tested in a while and the other one another one is a MSS so many security service providers people who are managing your sim and your IDs and all of these things more and more common and if you're not actually seeing whether or not they can see what they need to see then that's sort of a scary place to be we're going to talk about that some more in a minute

and then also pre-acquisition testing and then governance in general so let's get into the instant response part of this and I think that one of the important things about about this process is really I think is told busting stories penetration testing is an area where we really have to I think help connect with people because just at face value coming in and breaking things and then giving a report well we think it's super sexy business they don't understand where it fits in right so we have to start talking about some of the things that really matter like how confident are you that your controls are working as promised you have all these things in place when was

the last time you actually tested them and when you tested them best-case scenario they all worked as a hundred percent how many of the people in your organization we're actually able to tell what any of those alerts meant so one of the things that we see a whole lot when we come in for instant response is something Bad's happened and people have absolutely no idea what's going on so we get on site there's no logs who loves maybe have been rolled over and they have the last like eight hours of logs because yet they're logging that's been checked boxed but you know unless it happened in the last eight hours they're not gonna have any idea and then also not logging

the right things if you have no idea what you should be looking for you have no idea what you should be dealing with how do you know that you're logging the right things so what controls am I missing it's easy to go okay I've got the main check boxes here I've got my firewall I've got my IDs I've got my IPS what what are we missing and so this reminds me oh this is great segregation between the two environments right a lot of the pen tests that that we do we notice that people have not thought as much about segregation there is one segregating line and that is inside and outside and with a very

little control inside it makes it very easy for us to move around and one of the the biggest problems with security today is it's not a matter of when you're going to get hacked or or if you're going to get hacked it's you're going to get hacked how quickly can you detect it and how quickly can you respond to it and with those things without sufficient segregation how do you know where somebody's how do you know how they're moving around your organization and can anybody access everything after working in a university pretty much yes everybody can access everything but hopefully we don't want all of our organizations to be quite that open and so looking at it was

important how to segregate it efficiently while still enabling business to do what it does that is another problem that we see quite a bit not allowing are not considering the business requirements of what we do when we start making recommendations obviously we can secure everything but if you can't do work anymore then we're not gonna be doing this for very long so when somebody does tap attack how well can you actually detect it we've seen this a couple of different places recently but the defenders for monitoring things often are pretty in you dated any dated with data so figuring out what's important and what's not it's kind of tough so how can we deal with this one of the things that we

can do is we can actually run real tests against an organization and we can actually see what these things look like so when this is happening can you actually map attacks to log injuries can you actually map the types of exfiltration that someone's doing to the actual log data that you have and are the events that are firing coinciding with this because if you're not actually seeing the types of logs that you need to be able to detect an attack you're pretty much game over right so by actually sitting side by side with your pen tester looking at what's happening will heat the logs you're getting you can get a couple of different things out

of it one you learn a little bit more about how attacks are done so from a defender standpoint one of the things that defenders I think generally are lacking is understanding of how and why attacks work the way that they do lots of people especially in forensics they see attacks all the time they see the results of the attacks but sometimes the nuances are missed because why didn't somebody do this instead of this you know why did somebody go with the blind sequel injection here instead of you know an error message based sequel injection and sometimes it's those little things that make all of the difference and so being able to sit side by side with the tester and actually go

through that process and get that information is infinitely more valuable than just having a pen tester come in run a test give a report and then have the people go back and try to sort out their logs and see what they saw and sort of an another interesting piece here is can can you detect the bad things and so even if you can pick out what's important you may not really have a good idea of all the things are related and so that's sort of this goes back to just sitting down with your pen tester for doing some of this stuff figuring out what's noise in your environment figuring out what is legitimate activity is kind of

interesting for instance how many people inside an organization use PS exec for some sort of system administration tasks how do you tell the difference between when one of your system ends is using PS exec legitimately and when hacker is using PS exec maliciously what's your baseline for this activity where do these things come from can you detect when somebody's using it from someone somewhere that it's not supposed to well one of the things that I see pretty commonly is people are like well that was Bob we're just gonna whitelist this event so at that point that if that will never show back up again and so the biggest problem with sims is they're really noisy they they

tell you all the things that are going on so you just go ahead and mute all that stuff that's going on and then everything is smooth sailing from there right no more alerts we're good to go and so we want to make sure that we're alerting on the right things and they were not alerting on the wrong things so being able to detect that good versus bad is important so another one do people actually know what they're supposed to be looking for and one of the the funniest things that I see all the time is we go in we do with full knowledge here's our IP addresses were coming from you know all of this

other stuff and we say can you can you detect any of the stuff that's happened and they're like well we'll check into that and then like three weeks later I'll get a phone call and I'll say please tell me you're still pen testing us right because something else has happened inside the organization they can't tell what's going on because they don't have sufficient logging it didn't come from my IP address they don't think but hopefully you're still pen testing yes right and so not being able to see the good and making sure that you can actually see what the pen testers are doing the entire time being able to also be able to differentiate that from

what's going on outside of that is super important the other thing is pen testing isn't necessarily always the same thing that you're gonna get when an actual malicious person comes in one of the things that we recommend is actually going through threat modeling exercise in order to identify which are primary threat actors are likely going to be and try to emulate some of that behavior because they come in and run same set of tools that somebody runs for everybody else bone scanners that sort of thing you know it certainly detects certain types of things but it's not going to give you the same sort of alerts that you'd get if somebody was actually for instance installing remote access

Trojans into your environment and exfiltrating data those sort of things you don't necessarily always always see we were doing a pen test and during the closeout call the MSSP called and said on Monday we saw activity we think it was bad so this was during the out brief we were telling the client exactly what we're doing the MSSP had taken a couple days in order to actually alert them of what was going on but they weren't sure if it was good or bad activity so it was being left to somebody else and it was five days old so how much stuff can happen in five days right so another one is not just does your staff know what they're

looking for but whoever you have monitoring your tools do they have the ability not just detect what's bad and what's not but also act on it in a reasonable amount of time which kind of goes to to this point can you line their staff to follow the processes right and so there's two places here where I wanted to sort of tell some additional stories one of them was I was on on-site doing a pen test and I was on one side of the cubicle wall and the SIS admin's were on the other side of a cubicle wall I compromised domain controller and created a new account to see if they'd pick it up sure enough after about 20

minutes they picked it up and the person who whose account it was was at a doctor's appointment that ID used to actually create the new domain admin so what I listen to you for the next hour was them discussing how somebody created a domain admin account from the doctor's office and then when the person came back they thought that maybe it was good to implement the incident response plan so then from that point they discussed for another hour and a half whether or not they had an incident response plan and who they needed to tell so obviously this is something that had been practiced very well when to implement it how to implement it who to tell all of

that process wasn't actually sort of ingrained and so if you're not actually testing to see if people are following this plan how well is it actually gonna work if these guys didn't know if they had one what's the chance when it actually is implemented and the incident response process started that all the right people would be called in that all of the different areas that need to be involved would be involved and then there's sort of the other side of this we're on another test I was doing I was Co teaming with someone else and one of the assisted mens we sort of knew it was going to be bad when we walked in and

during the initial briefing suddenly the the set of men's views on their security posture changed they would told security everything was awesome and then suddenly well maybe maybe there are a couple gaps here and we don't think they'll find anything and then as discussion continued it got worse and worse well so the system has decided that the the best course of action during this pen test was to watch what we were doing so they had put port monitors where we were so that they could see everything we did and then that way they would know as soon as we got into a system and then they could fix it right then and then that would

not be a problem they were gonna you know get through this this stunningly so as we start working through systems we we capture one of the system men's accounts and start using it it happened to be the sysadmin that was watching us through this process so what happened next to that was a series of temper tantrums including some actual honest-to-god tears and to make sure that we could get no further I went and rebooted the main controller because that was the best thing to do to keep us off of the domain controller so as he kept changing his password we kept getting it back and using it again and so instead of implementing the instant

response process he bounced the DC so obviously um maybe at least he was responding I guess but not having a standard instant response policy in order to be able to to work through this means that you have no idea with the actual impact your organization's going to be it could be bounce the DC and who knows what happens after that right and so making sure that your people know what process is to follow and know what they need to be doing is it's really important so we're gonna to go over this point sort of where I think that this should be headed and then hopefully you guys will have some additional discussion so for the most

part nobody does remember the things that you did well they only remember where you failed and we see this all the time with the stuff that's in the news right now the things that are in the news are not because somebody got breached but it's all about how they're handling these breaches and so when you see things like oh yeah a vendor presentation showed that we've been compromised for almost two years things like that aren't very good stories for the news being able to come in and actually be asking the right questions about things like do I have sufficient controls can I can actually detect when people are moving from place to place what do I have as

far as the ability to detect normal activity versus malicious activity so it's not just necessarily the controls but how well those controls are tuned people are another important piece so how many people here part of instant response teams okay how many people here are pen testers how many people here awake okay it is early thanks for coming out by the way I didn't mention eyes it it really is early and so making sure that not just the controls have are able to detect the right things but the people know how to interpret them and one of the biggest gaps now is the people gap people are interested especially for blue team unfortunately not looking at hiring the people that

have the most experience and they're the best but there's a big push just to finding people and more people does not necessarily mean that you'll detect more things it just means that you have more people that need more training and so making sure that you have the right people in the right places with the right training and I think this one's always sort of a stickler but I think the right experience is to you know a lot of the people who are coming out of security programs right now you know when you hire an entry-level security person a lot of the times they're missing that experience and while they may have a degree sometimes when you

have somebody who's actually come through I've been assisted man I've been this I've been that they've seen enough different things that they know how the world is supposed to work so you get sort of a different experience out of those people so just having a cissp or just having a degree in computer security well yeah the CISSP is you you're not drinking I was hoping you'd be drinking um so all of that stuff is certainly beneficial but making sure that you have the right people in the right places is one of the most important pieces so in them process so when when we start talking about pen testing and some response we have a couple different

things that we do the first one is we work on helping people build up their security posture so you can see the right things right if you can't see the right things then you're kind of screwed so what we start off doing is things like threat modeling so that people can figure out what they have why it matters who might be interested in it let me start looking at what the controls are around that and we start testing those controls and if there's an insufficient controls the controls aren't working is is we expect them to those sort of things then we look to see whether or not people can actually determine what's going on and so that's where we start

actually sitting with the people who are looking at these and doing the training that they need to be able to detect one problems are happening and so obviously listens sort of tough it can't just go in with pen testers and say oh we hacked some stuff you should have seen some logs right so having the right people with the right training to help with this is important so this is one of the places where I actually recommend experience instant responders work with the pen testers in in training the people who need to be doing this monitoring to be able to see the things they need to see but also the instant responders are going to know exactly

what logs they need to be able to pull this off and then also establishing baselines of malicious activity a lot of the tools out there will learn right well if you're already completely compromised learning it doesn't really help you very much right so being able to identify exactly what is is good in what is bad inside the organization I think is super important and being able to figure out exactly what people need to be looking for so that they are making the right alerts quiet and not the wrong alerts quiet false false positives are super annoying false negatives or the ones put you in the news right so we want to make sure that the the number of false

negatives are limited and play said we've seen this a lot of the time recently where you know a large store retailer that was breached actually saw the alerts and they just didn't know what to do with them and so it turned out that that was all of their cash registers being compromised and so obviously not a great news story and then when you start talking about due diligence if your people can't actually detect bad information or bad activity from the information that you have have you really done sufficient due diligence and I think that as more and more lawsuits start popping up around this and people start getting denied more for their breach insurance that this is going to

become much more of a big deal another area where this is becoming a big deal is the SEC SEC is decided that they are going to start prosecuting c-level executives that were negligent in information security practices so if they knew that there were problems and didn't do anything about it and it led to information disclosure about people then making now go after c-level executives and they can do everything from finding you to stopping you from being a c-level executive or a board member from a company again and so this is something that they are more actively pursuing and obviously with the government right now there's a couple places where they're pursuing it but it's not a fast process so there's not a

whole lot of case evidence to go based on this this this and this this is what you should be worried about but this is something that's coming and they're serious about actually following up more on and so the other one is making sure that you're actually communicating with your consultants so regardless of who you get for to do your pen test and who you're working with make sure that you have the business discussion about what you're trying to accomplish a lot of the pen tests that I see out there somebody is like well I've got this many web servers I've got this many file servers I've got this many this kind of servers we want to pen test oh why why do you

want to pen test and so asking that why question so that you can help support the business objectives of the organization is is probably the most important thing that we do because if we're not supporting these business objectives and we're not telling the right stories then we're not getting to the right people and especially when you start talking about doing things like making sure that people's processes and those sort of things are working correctly that story is extremely important because that's one of the things that people worry the most about is I've invested all this money how do I know that it's helping me at all and this is a great story to be able to tell we can come in we can test

all this stuff we can make sure that your people are seeing the right things we make sure everything's working correctly so that you don't have to wonder you can actually know right so those the the main things I wanted to talk about today hopefully it was hoping you guys could have some additional discussion as well so here's my contact info anybody have questions comments critiques yes I was talking with the consultant a couple of weeks ago and they had mentioned there was a health care company that basically on their cyber liability insurance when they had filled out their application they basically weren't completely honest and they had a breach and obviously they you know what did they solve a liability

carrier and after they did their review of the incident they actually went back and insured the cyber liability carrier was actually suing the organization because they found that they were actually no so it was okay yeah did they give you a name no no okay well in that due diligence is a big thing right now one of the a lot of places that allow self attestation that you're doing all the right things it's kind of interesting because even if you are being as truthful as you think you are and you don't actually know what your state is you you may be misleading enough that they'll reject that so making sure that you have all of your

stuff on point you can prove due diligence you can do all these things before you actually go out and get the the cyber insurance for for breaches is something that's really important one of the things that I hope to see more from the insurance providers is actually doing some requirements for that so that if they are insuring people that there's at least in a set understanding of what's there because right now with the general quality of the pentose being all over the board if you go and get joe bob's pen tests and they're like yep everything's green then are you really still do doing diligence if somebody else comes in who is actually somebody who's skilled and

finds tons of stuff which which is right you may have thought that you're doing the right thing by giving joe bob's pen tests when actually you're you were really far off so it's it's something that is an area that's kind of uncharted right now but I think it's gonna become more important in the future anybody else a good question cool that's 9:25 okay thank you so yes sound guide has a question so let's step back I think organizations that have our publicly traded that the SEC can go after or have enforced their rules of being go down a level private organization lots of cloud-based services that you know their start up they're getting some traction but

they're they haven't quite grown up yet how do you get an organization like that to actually have the tools to even have the logs it's so easy to have a hundred Amazon instances but no one is doing a login server so if you tell them go buy this enterprise tool they're gonna say no so where do you steer that group of people to when there is no sysadmin other than maybe the developers or some freelancer they might find if they have a problem so that's a good question and well and there are people that obviously like Josh that help people with that exact issue but you know at the end of the day it's what's important to people

right security is not important to you you probably will be at some point and most of the time you hope for people that that's not when they're in the middle of a crisis right and so especially with startups what's important right now is getting a product and making money and then figuring out the security stuff later because they don't realize that when you actually have to go back and try to figure how to bake in security later that they're basically screwed right I mean when you start talking about having to add security to something that's inherently unsecure you've got all sorts of problems and the other piece of that is not having the right people because if

you've got the right people then those right people or actually when they're there DevOps saying everything making sure that they have all of the different logs that they need they have them at least in a central repository and it's not hard to get tools that will help you deal with some of that I mean everything from Breaux is free for IDs there's lots of different free things that people can use in order to do some of this OSM the sort of things all of it is possible is just having the right people with the right understanding to do the right things and I think it's completely a priority issue small people don't think about security because that's not what

they do and until something happens it's not an issue so I think that the the other piece of that is making sure that people understand why they're at risk and that sort of thing so that they can be something that they're at least thinking about a lot of people feel like they're too small for attackers to be interested in but more doing instant response the people who are most negatively affected by a breach are the mom-and-pop shops when we come in into instant response $500 an hour the amount of money that a mom-and-pop shop or a start-up can take with $500 an hour of investigations is really small and so a lot of these people that we see that either have to

choose never to accept credit cards again or to go under and it's sort of just a bad sort of deal so I think we need to do a better job as a security industry of making sure that people understand there is no too small to get hacked but good question happy else cool thank you guys very much [Applause]

you