Pentest Pains

Well, you can find him all over the place on all of this. This is Chris Trainer. He's going to be telling us all about how pentesters are absolutely perfect and nothing ever goes wrong. >> That's what you nailed it. Yeah, that was the abstract. Exactly. Word. >> Absolute abstract on this. >> Yep. >> Seriously though, Chris is a trainer. He is a pentester. He's been around for a long time and a very smart person. I'm glad to actually be able to know him. Let's give it up for CHRIS TRAINER. >> YEAH. [applause] ALL RIGHT, THAT WAS too much good words. I said just to say pen tester at Black Hills. All the other extra stuff.

There's definitely way more. Okay, >> take the mic out of the holder. >> You want me to hold this or is this a fine? >> It's for video. >> Ah, okay. So, we got to do this. >> All right. I'm used to just kind of yelling in my classes to the all the way to the back. Uh so I'll do the mic and try to keep that in front of my face best I can. All right. Thanks everybody for coming to Pentest Pains. I am Chris Trainer. I am a pentester at Black Hills Information Security. I also am an author and instructor for two classes uh that I teach through anti-ciphon using my own Ridgeback infosite company. uh offensive

tooling for foundations and offensive tooling for operators. I got certifications. You can like them, you can hate them. We can have a threeinut discussion about them after that. It's kind of kind of difficult to keep talking about certifications and I hate talking about myself. So, I'm going to try not to get distracted by the really fluffy bunny outside the windows hopping around in the grass over there that I can't stop looking over at now. But pentest pains, why is this talk needed? Right. So, I've got my fancy laser pointer that's sort of coming up. Uh, this talk was born out of years and years of painful engagements that occur either from the pentesters side, we are

the cause of the pain, or from the customer's side. I really, really tried throughout this entire presentation to not make it sound like this is a 20 minute gripe fest about customers. So I had to throw in, you know, just a handful of things that pentesters do wrong as well. So if are is anybody a pentester in here now or red teamer does offensive engagements. So a little bit anybody blue team defensive side who's a manager? Anybody Anybody manager who hires pentest companies and stuff? Okay. All right. You guys are perfect. Everybody else has problems. Um, [laughter] and you can come to our booth if you need pentesting. Now, we're trying to get better with this whole talk here.

Uh, painful engagements have a lot of consequences on all sides of the table, right? Some of those things obviously are less secure environments, loss of revenue, um, reputational harm for both sides. If I do a really bad job on your pentest, I'll bet you anything that John Strand, the founder of the company, is going to know about it. and he's going to call me about it and we're going to have a conversation about it. That's not only a reputational problems for me, that's a reputational problem for the company that I work for. Uh if you do a really bad job as a customer and you are a real pain to deal with, most pentesting firms I know that we do, uh

there's always an opportunity to say, "How was this customer? Should we do business with them again?" Or were they just an absolute pain to deal with? They were challenged absolutely everything. they threatened you, they were adversarial the whole time through. And that's a real quick way to get yourself blacklisted on a pentest firm's uh list. Worst case scenarios are litigations, sometimes terminations. I've seen terminations happen on two sides of the of the table with pentest and with engagements where there are uh pentesters that just handle a situation absolutely horribly. They have a terrible ego about them. And the same thing on the other side. You have an IT manager who spent 20 years and his

infrastructure is his baby and it's beautiful to him and you come in and you write a terrible report and he gets real angry real fast and starts bad mouthing you, the company, the report and to everybody because he feels like it's his job on the line. And in some cases that is true. This talk does apply to everybody, not just testers, not just managers. I'm going try to hold this away because I feel I hear that coming back. Uh if you are a sock analyst and you think I have nothing to do with pentest, I don't have to deal with it at all. I can't cause any pain in this environment and they can't cause me any pain. You'll be very

mistaken in that uh arrangement. I've got on the phone with sock analysts who have shut things down while we're in active testing and they cause a lot of problems too. So this is not just for the tester, not just for the point of contact. Uh but most of all keep in mind that most of these engagements they go very very well. These are the outlier scenarios that I'll kind of talk through a little bit. So getting your something that's having feedback >> stage on the front of the stand >> front of the stand. Okay, I'll try to move over here a little bit. A little bit of feedback. All right. So first and foremost to to ensure most engagements

are not painful. The easy answer to all this stuff right up front is to have good clear and clear communication and good expectations from management right up front. The best way to do this types of thing is with a rules of engagement call which talk about very briefly uh in the next slide. But this all stems mostly from preconceptions and misconceptions around what a pentest actually is. Right? I'm not coming into your environment trying to bypass everything and anything that you've put in front for defenses. I will ask for some concessions. Most times, more times than not, you will come into an engagement where I have only five days or 10 days. A typical attacker will stay

in your environment for months at a time. So the real real driving point around most of these issues is that we are working around a time constraint. We want to get you the best best amount of of work and identify the best or the worst problem so you can address them. Understanding the difference between a pentest, a red team, purple team, social engineering and fishing engagements is very common on a customer side of things. And even on the pentester side, there are confusions between each one of those types of engagements. There are entire talks you could talk you could have that compare and contrast all those ones. They are all different names. They all overlap a little bit with each

other, but they have different goals. The goal is what you should be focused on. The goal of a pentest is to find out what systems you have that are actually vulnerable and actually exploitable. The goal of a pentest is not to make sure that you can see me. I have five days. I'm going to be the loudest thing in your environment for those 5 days. You should see me. If you don't see me, you have an issue with your monitoring, but please, dear God, don't try to shut me down during that time. An overzealous sock analyst has locked things down. an automated uh lockout of systems has occurred on many tests and that winds up

shutting me down for hours or days at a time and that is very valuable time to lose. I mentioned previously rules of engagement call. This sort of thing happens before the engagement. You should be having this weeks before. It should have the point of contact that is there for the test. Meaning on both sides, I'm your tester. You're going to see and hear me before the test ever begins. I'm going to go through the rough outline of how the test is going to go and I'm going to make sure you have a good understanding and your expectations are clearly managed. It's the hardest thing in any job that is a desk job. I won't talk about road

construction or anything where you're doing work out in the hot sun. Those have their own challenges. But for our jobs, the hardest thing is expectation and communication. So, rules of engagement. Actually, on June 11th, a couple of our guys are doing a a webcast on all about rules of engagement for an hour. So, if you don't know what this is, that would be a good thing to sign up for uh and check out. You're going over common things like the motivation for tests, the confirmation of testing dates, what is in scope and out of scope of your systems is critical during these conversations. I've had tests begin and then somebody was out of the loop and when I send my hey I'm

starting the test today just a reminder you're going to see me I'm testing this system they come back 2 hours later after I've started testing and say that's not supposed to be what you're touching and then they have to go off and have a half a day conversation around what's actually in scope and what did they actually pay for. All of that needs to be ironed out in a rules of engagement call. All the involved parties need to be on that. Another thing that I'll always talk about in this is if there is an emergency during this time, if you are the point of contact and you have a family emergency and you're the customer, let me know. If

you are the point of contact and you're flying to New Zealand from California three times that week, let me know. That is a real scenario. There were real problems on that test. They didn't tell me they'd be in the air and offline for half of that time and I couldn't get anybody to answer anything. Uh if I find a a scenario where I have I've identified that there is a indicative of compromise in your environment, it's a stop work order for me immediately. I pick up the call. I'll call you. You'll know I'm stopping. You'll know the situation. No testing will continue until your IR is complete and we get a green light. But everybody needs to be

on board of what that emergency plan looks like. How's the communication going to happen? Who's the backup points of contact? And who can you escalate to on both sides? Some very common client-driven pains. Some very common client-driven pains here are around inaccurate scoping and inaccurate uh lists that they give to attack. This is a weird one with not adhering to RF uh RFC 1918. Who knows what that is? Some people have an idea. If your organization isn't adhering to that on your internal network, there's a chance I'm attacking one of your systems and I'm accidentally routing out to the open internet and attacking a public IP address because you didn't use reserved IP space. There may be scenarios where

that public IP address is another nation state that could go poorly. [snorts] Another one there for environmental preparedness. Always be ready for the start date. If you're a customer, you're not ready for the start date. there'll be a finding on there uh in the report saying so in a nice way, right? But if we're losing a lot of time, it's your fault. They will be called out. This is a huge one. Don't be adversarial. Don't be taunting to your pen tester. Don't demean them for their age, experience, sex, anything. It will go very badly for you. If you challenge the person who's meant to tear apart your environment, if you call them a junior pen tester, I

guarantee and has had this, we there's a tester that will immediately get AD on your environment within the first two hours. And that has happened. That customer also might be the type of customer that threatens litigation within the first day of environments, right? I don't like what you're doing. This isn't allowed. My baby is beautiful. I will sue you. some tester driven pain points on there. Uh if you get a scope and you're not verifying it, somebody hands you a list of systems and you just blindly start attacking them on day one and you didn't do any due diligence of that external scope. Most of most customers are bad at configuration management databases and knowing what systems they have and

they're providing you a list that was compiled two years ago. Those systems could have gone offline. Those systems can no longer be theirs and you could be attacking the wrong person. Always ask in a in a pre-engagement if you have uh customers that operate in health care or industrial control systems that after you get the scope and after they've given you what's out of scope, ask again, are any of those systems critical to human life? I've tested a lot of is ICS places. I've tested food manufacturing plants. I've tested hospitals. I've tested companies that they don't think they operate anything where it can affect human life. and I absolutely can. All those systems should be removed from scope. If one of

those go offline in a hospital, that's a really bad day, not just for the company, but probably for a patient and a family. Definitely don't run DOSs exploits. That one's pretty good there. If you're a tester and you're struggling, don't have a huge ego. Ask for help from your other testers. I have questions all the time. I've been doing this for years. Uh Ashley, who's at our booth out there, I've asked her dozens of times, "I can't remember where this is. I can't remember what this is. Can you help me out?" And she'll jump in and she'll help me out. All right. I forget things. I write them down so I don't forget, but I always

forget something. Always report as you go. As a tester, you're not reporting as you go. What happens if my 2-year-old brings home a sickness from the daycare, which happens every 9 to 12 business days [laughter] and lands my entire family out of commission for several days? This happened many times over the last several years. Uh I've been able to hand off my test immediately to another tester and they can just pick right back up where they where it left off. It's not about me winning the lottery or getting hit by a bus, but the company and the customer should be taken care of and it should be in the report as to how far you got. what's next and what they

should do. Always have a report that's actionable. If you deliver a report as a as a pentester and they can't actually fix anything off of it, I've gotten these reports many times on my own. Uh where I'll get a report from a uh I've been on an internal pentest team and I get a report from an external team. I don't want to say it was complete trashing garbage, but it was a vone scan masquerading as a pentest report with no verification checks being done, no attempts at exploitation being done, and most of it within two seconds of looking at it is completely void and not valid. They didn't do what they were supposed to do. They charged you tens of

thousands of dollars and didn't do the job. If that report's not actionable, send it back. I'll say on the on the other side of that if you receive a report as well and there's context that should be given to supplement that report uh always have a debrief call between customers and pentesters I give you the report I say hey within the next 30 days or so you want to talk through it let's make sure you understand what I meant by this I don't expect you to read the 65 pages of methodology and deep technical steps and every command that I wrote and where it was and screenshots and all that but I expect you to

understand the finding things and I expect that how I wrote the uh the executive summary and the actions and the strategic guidelines for your executives, I expect that those can be consumed by your bosses. So if they're in a format or they're missing some information, always have a debrief call. Always walk through those environments together. I have no problem. I've removed findings from my tests. It takes a hell of a lot to convince me it should be removed. But I have made mistakes and I will not have an ego about that. If you have a pen tester that just stamps their foot down and says, "Nothing's changing with my test. I pressed the save button and delivered the PDF. It's

yours now. Anything wrong with it is your problem." That's not a good sign. As a tester, there were a few offensive people in here. Has anybody ever tried to attack a system or an environment that they'd never done before? I hope so. I hope every hand that was up before goes up because you at some point you didn't know everything you should have been attacking stuff. It's perfectly reasonable to take on a test type that you're not thousand% comfortable with but you should always have somebody behind you who is comfortable with that. I know for for Black Hills Infosc before we allow a tester to be solo on a test in an engagement they have to go through

uh anybody ever heard the term murder board before one person. Uh so basically a murder board is we have you tail another tester on that type of engagement that you're not particularly strong in yet and then afterwards we'll sit down a group of a few testers and we're going to hammer questions at you like we're your customer. Right? One of us is going to be adversarial about it. One of us is going to be challenging your technical aptitude on it. One of us is just going to be I don't understand anything. Explain everything to me. Right? So a few different scenarios. You have to be able to handle every type of customer that you run across. But most

importantly, you have to know what you're doing on the test. Communication, again, reappearing again in the slides here. More important than communicating, if your customer sends you something during that engagement, be responsive that day. Nothing leaves a bad taste in a customer uh experience more than just a tester who's in your environment trying to break everything and then suddenly they vanish for 4 days while they're in there and they don't respond to you until the test is over. That's not good. Due dates. Who does any social engineering? Social engineers in here. A couple. You guys probably have to deal with this more than most of us. Uh, but knowing where to draw the line on being

unethical or immoral on your attacks, right? I know that there are a lot of scenarios. I've done I've done spear fishing type of stuff. We have people do social engineering. If I ever got a call, if I was a customer or I was somebody not involved at all and I got a call from somebody who was social engineering me saying that my child was endangered or their well-being was at risk and that's how they were trying to elicit an emotional response from me, they're going to get that emotional response and then when I find out they're lying, they're going to get another response that's very much more emotional. Know where to draw the line. Don't ever ever threaten the

health, well-being, or anything like that of your customers. All right? You can bad attackers do this, but you should absolutely not. So, when things go wrong on a test, which happens often, I've told some stories before uh out there. I have a series on the Black Hills webcast channel uh called the Illustrated Pen Tester, which is just short stories of security. Um on some of those, I tell a bunch of these examples. One is if I find an indicator of compromise, it's not me inside the environment and we stop that emergency plan. How do we handle it? A lot of the problems that we see when things do go wrong is that immediate switch from a customer or a pentester

who was doing things very well. They were on the same page and helpful, but as soon as something went sideways, they got very defensive on the engagement to the point to where the test sometimes just has to be called off. Right? That doesn't help anybody. We're all on the same team for this on both sides. Again, I'm repeating again because ego has been such a big problem in a lot of these pain points. If you have an ego going into a test, you're going to have a bad time eventually and you're going to come up with somebody come up against somebody at some point where they're not going to tolerate your ego. Right? Every customer has the ability to say Chris

was a terrible pentester and I have every opportunity to say they were a terrible customer. Right? If I if a problem comes up, again, like I said before, that I get feedback that I did a horrible job and I was also an ego-driven maniac during the test, that's not going to be a good conversation that I have with my boss next day. Professional and and and good communication always good. Uh document everything you're doing at all times, right? You find an issue, you find a vulnerability. It's happened to me on a lot of tests. Something's wrong. I was able to exploit it. I didn't take the screenshot. I wrote it up. I didn't take the evidence. I didn't get the

screenshot. I go back a day later. I can't do the same thing. It may still be there. Something may have changed in their network configuration that's preventing me from doing this stuff. Uh so always document as you go. Know that you don't have to be the bad guy in that rules of engagement call and everything. You should be establishing on both sides a point of contact to escalate to. There have there is a story out there about uh there was a very adversarial IT manager again who thought his baby was beautiful and he thought that it was a challenge to the pentester to prove every step along the way with no concession and no help or no communication between them

and then he was saying hot you didn't get it you can't hack me you're not good enough right turns out that IT manager his name was mentioned in a password that was cracked on the engagement and it wasn't mentioned in a flattering way his other co-workers did not care for him very much. There were some expletives mentioned and then we uh it wasn't my report, but that password did appear in clear text in the report. [laughter] If you want to be spiteful and hateful, we can do it back and we can have it in a discoverable document for litigation later. Uh but you don't have to be the bad guy. If you're the tester or if you're the

point of contact and it comes up against a problem of legality or a problem with the contract and the scope or money or any of that stuff, step away from it because that shouldn't be some conversation you're having. That's going to be a really bad time. That was 20 minutes. That went by pretty quick. Um, I have a whole bunch of other stories out there, like I said, that highlight pen testing uh pains in hourong formats where there's like five or six stories at a time. Uh, they're all out there on the on webcasts and stuff with Black Hills. They're listed out there on my Ridgeback Infosex site, too, to find that. Uh, but does anybody have any

questions or particularly painful scenarios you want to share? Just about your webcasts, are those like your training videos or are those on your YouTube? >> So, the webcasts, they're on uh they're on YouTube. They're not my training, not my training videos. Um the training classes you can get through anti-ciphon, my two classes are on there. If you go to Ridgeback infosc, those are just my classes that I offer and then they'll direct you out to anti-ciphon and stuff like that. But those webcasts, there's a handful of them uh that I've done over the last several years. Those are all on the Black Hills uh YouTube page. >> Just going down the EJP class. >> Oh yeah. Yeah.

>> Do you do physical too? And what's like your worst uh pain point scenario on physical? Like John Strand mentioned he won't do arm security type thing. >> I don't do physical uh pentest personally, but a number of the stories that I've told I aggregate them from other testers that I know directly so I can verify the information. I've read the reports. I know it's a true story. Uh, I've heard some some weird ones with physical stuff. Some of the ones that go bad is if a tester does not have in their back pocket the get out of jail free card. Uh, there's scenario with a courthouse a while ago that Darknet Diaries did um where the testers had

every piece of documentation they needed to get out of jail and the sheriff just wasn't buying it and they wound up getting booked and charged uh with a felony trespassing, breaking and entering. And if you ever try to go for your clearance for the government, just being charged is enough to get your clearance blocked. So you can't do any work in that field. There does not have to be a conviction and it's very lengthy process to get that reversed and get your uh record expuned completely. But there's been scenarios where there's overzealous uh security guards. Uh I did tell a story where we had a tester get in and we never send one tester on a

physical engagement. It's always at least two. One was sitting in the car and one got into the building and the one in the car was challenged by the security of the building. And you're supposed to lie as long as you can. And they asked him like leading questions. Are you here alone? Yeah, I'm just picking up my friend. He's in there. He'll be out in a minute. He works here, but I work down the road. And they're like, have you ever been here before? No. They actually had security footage that could match license plates and they were able to track back before they ever challenged him that that license plate of that rental car had come in two days

before that while they were doing recon on the property >> and they saw him get out of the car and go into the building. So yes, >> um do you have advice for how junior paners campus could contribute to an engagement um but also learn for themselves? >> Yeah. Uh I'll stick within the example of of the physical security stuff right now. We had I wouldn't say I I hesitate to call anybody junior because everybody has their own expert experience and backgrounds. Uh but we had a more um lower in the age bracket uh tester who also she had a younger physical appearance and we had a physical engagement coming up. So we partnered her with an older pentester and they

went on the physical engagement as a fatherdaughter facade. So as they were walking into the building they were talking as if they were father and daughter. He was taking her to work to show her take your daughter to work day type of stuff. And they played on the psychological uh aspect that people are much less likely to challenge you in mass. There are few people that are jerks to challenge you if you have a child with you. if I tailgate you in and I'm telling her about my job and you turn around and challenge me and make me look like a jerk, you're risking looking like a jerk yourself more. Um, and they did get in on that engagement. So, in

the physical world, you can do that in the junior stuff for the more digital type of type of stuff as well. Uh, everybody has their own experiences and backgrounds and stuff, right? I don't know certain things on AD. Um, I ask people I know uh Alyssa Snow who's a great pentester at our organization. She's younger than me, but she is much more knowledgeable in those areas because of her background. So, make your background known in the organization so people can leverage that. Yeah. Anybody else? Great. Anybody got any questions or talk stuff? I'll be out there or at our table. [applause]