Pentest Pains - Chris Traynor

Can you hear me now? All right. This is going to be weird because I never usually hold a microphone up to my face. Uh, okay. Cool. Good. Good. All right. All right. Uh, first off, thanks for being here instead of in a movie theater watching the 20th anniversary release of Revenge of the Sith. Um, I just found that out yesterday and now Sunday night at 10 o'clock when I get back home and actually go through it. But this is Pentest Payes. I'm Chris Trainer. I am a penetration tester for Black Hills Information Security. Uh, I'm the owner of Ridgeback Infoset, uh, which exists only so that I can create classes for anti-cipyhon and teach through them. I have two offensive

tooling foundations and offensive tooling for operators. If you're interested in those things, see me afterwards. But are there any like professors or teachers, people that educate in here, see me after class because I can send you guys some cool stuff. Um, one of the things that uh Black Hills infosc likes to do is we like to just kind of like give a whole lot of stuff away. Uh, I have a small amount of things in my bag. Uh, so I'll step outside of there for a little bit. If people want like little magazines or back doors and breaches like card decks and things, I have a few of those on hand. Only what I could really have them shipped and I

could fit into a backpack because I didn't want to carry it on a plane. Um, so home test pains, why is this talk really needed? Uh, the most basic answer is just so that we can all kind of get better. I've tried really hard when I was creating these slides to not make this like a 20 25 minute gripefest around I hate it when customers do this and customers hate it when we do this. So hopefully it doesn't come off that way. I also rewrote these slides at like midnight a day ago right before I did this exact same talk yesterday in uh Harrisburg, Pennsylvania at their bides. So um I hope it just kind of comes together

nicely. So mainly when we have problems with pen tests uh and and things that make it very very painful for us for engagements uh primary drivers why we want to know what to do what not to do is because when you do get into a contentious situation on an engagement it winds up having a lot of negative effects for both sides. It's not just the pentest firm. It's not just the customers that get affected. You can have some serious issues including litigation. You could have some serious loss in revenue. Both sides can have loss in revenue because if you wind up not getting a full pentest because you were a bit of a jerk to your firm and then audit time comes

around, we couldn't get on another pentest firm's schedule because it's Q4 and you didn't plan right, that's going to come back and bite you because you may have some some fines due there. Uh, this talk's not really targeted at customers or pentesters. If you are an ancillary position like a sock analyst, you're not ever directly impacted by a pentest. what's going on. You do still have a role to play. Uh most of the time when you're doing an engagement for a pentest, you the organization knows you're there, but the sock analysts don't know you're there. And a lot of times what gets called out is a sock analyst gets a little overzealous or policy or procedure is a

little too restrictive. And there's been many times on a pentest where suddenly I can't reach anything because Billy the sock analyst decided to shut down all communications from the implant that we had deployed on your network. Um if they don't have a proper roll up of of escalation that can be that can be very painful. So this is good for ancillary positions. Who's a pentester? Who's a person who hires for pentest firms that make decision? you guys can see me afterwards, too. Um, but through this through this talk, make sure you remember that I'm I'm pointing out some of the negative things that happen, but most of the time engagements go well. Nothing bad happens. Both sides are very cordial.

Everything goes off without a hitch. It's not too terrible. the hardest thing to do for any any office job. I'm not going to say anything about like food service industry, which hopefully most of you have spent some time in because if you haven't, you're missing out on a big piece of painful life experiences. Um, the hardest thing to do at an office desk job or at home office is communication, expectation management. It's if you can do those two things, you're going to wind up resolving most of the stuff that we talk about here today. you're going to get a lot better at your job. Even if you're saying, "I can't do something." Just proactively telling your point of

contact, "I can't do this thing because XYZ," even if they get upset, it's going to be a whole lot better than them finding out you didn't do something when you issue a report. A lot of the problems that we that we run into and make pentest painful is because people have the wrong preconceived notions and preconceptions about what type of an engagement they actually purchased. Right? If you hire for a pentest, don't expect the pentester to go through a lot of leg work to actually try to be evasive or hide. Pentest tend to be shorter engagements than say a red team where they're going to be evasive and spend time trying not to get noticed. Uh but

absolutely if you do have a pentest and notice the activity personally I would love for you to tell me that because I will put that in my report. Even though it's not a oh no, you found me. I will give you a 100% gold star in the executive summary for having some kind of monitoring in place. You did see me. Please don't stop me. But I would like to still know because I like to call out that you guys are doing some things right. So getting proper definitions from your from your pentest, knowing what you purchased as an organization uh and what to expect and what concessions need to be made for the different types

of engagements is very important. Another thing that can really help you nail down any of these pains before they ever start is having a rules of engagement. Anybody familiar with the rules of engagement? A little bit. Okay. So, basically, this should be a very short, it should be a a very informal conversation that you have either as a pentester directly with the point of contact that you are going to be working with. You're going to be nailing down a few specifics. It's just a handoff call rule. At that point, as a customer, you've probably only interacted with a salesperson or contract person, a money person, things like that. But you should know the person who is legally trying to

break into your stuff. You should know their face. You should know their name. You should have every piece of contact details you can during that time. And they want to know you. If you hire a a pentest firm and you never see, hear, talk to, get communications from the person that's going to be spending the next 5, 10, 15 days breaking into your systems. That's a bit of a red flag. A yellow flag, right? It's not the worst thing, but they should be reaching out to you and the pentester for that engagement is responsible for making sure that communication happens. You're the customer. You paid them up. They should be doing right in this rules of engagement call.

You're just confirming dates, making sure what's in scope, most importantly, what is out of scope for the engagement. Uh defining a communication cadence, especially if something goes wrong. Personally, what we wind up doing at at BHIS is if we find a critical issue, our default stand for critical issue is we notify you immediately. You're not going to get an email. You're going to get a phone call. If you don't pick up in the first 30 minutes, I'm going to call again. If you don't pick up after an hour, I'm going to call your boss or the other secondary contact. And if that person doesn't pick up because of me, not for my lack of trying. I escalated

on my side. and people above me who decide if I have a job know I did my work and they're going to try contacting your boss because it's a critical problem that needs to get discussed now that sort of communication also will come into play if you have an indicator of compromise meaning I found something going on in your environment that's not me and it doesn't look like it's you either and this does happen it doesn't happen too often hopefully for testers and companies uh but when it does. It's important to have a clear emergency scenario plan uh where work stops. If I find that you have an outbound connection attempt coming from a power

controller for your hospital's backup power supply, I will stop. I will call you. We are not going to do another thing until you absolutely tell me I can proceed and you've done all the forensic work. Yes, that's a real scenario. Um you also want to make sure you're nailing down any high-risk specific actions. So high-risisk is kind of there in quotes because high-risisk is a different definition for every organization. It's not really something that you can say uh applies across the board of what is high-risisk. I can ask for 50 calls, 49 of them will say absolutely do password attacks against anything that you find any authentication forms. Try anything that you want. I'll get that 50th customer on

the phone and they say don't you touch that. Don't do that. because if we get one account locked out, we're going to have a hell storm from everybody else in the W department. Some examples of our client-driven pains mostly resol revolve around scoping. Honestly, for for a customer, your main job is to tell me what I can attack and what I can't attack, right? So if you're giving me an inaccurate or confusing list of scope systems and confusing meaning that you provide both your incope and out of scope in the same document right next to each other with no line separations, no headers or anything like that and you give me a bunch of business specific jargon as to

what the systems are called that I'm not privy to knowing those definitions. That's confusing as hell. I'm going to come back. I'm going to ask questions. I specifically put on here cider notation and people familiar with cider notation a little bit a little slashy in 24s or 16s or things like that. I never like cider notations whenever I get anything. The only time I accept cider notations just kind of default on an engagement is when I'm dealing with a company that has actually told me we don't know what we have and we don't know what we don't know. We just know that we operate in these IP address spaces. What I do in those instances, I usually take a few

precursive steps before an engagement begins to identify live systems for them. But I'll never feed a cider range into one of my offensive tools. Ever. One of the things I talk about in my in my classes is that different tools interpret cider range notation differently. If you gave me a 1 19216856.1624, one tool is going to think you're meaning one singular system. Another tool is going to think you mean 16 and up in that subnet, and another tool is going to think you mean the whole SL24. Cider notations are bad things. Uh, duplicating systems and lists, it's weird, but it happens all the time. outdated configuration management databases or some outdated lists from years ago causes us a great

deal of pain for getting started. Um, something I see more often, universities, things that have been around since the dawn of the internet are not adhering to RFC 1918. Who's familiar with that? It's basically what what tells you what is a reserved IP space for internal IP addresses. Uh, the main reason why I don't like it when a company that's hired us for an internal pentest on their internal network and I see a bunch of public IP addresses listed out is because one slip up in the routing of traffic means that I'm going outside of your network and I'm attacking some other system on the open internet with that IP address. It's happened. it's happened where maybe

those IPs might be um let's say hypothetically tied and and assigned to foreign states and that could be very bad. So adhering to RFC 1918 your organization and if you don't just give me a damn good reason why you don't. I know it's hard to go from a system that's been around for 40 50 years and try to do that retroactively in a live environment but it's definitely worth it. One of the biggest things for a client-driven pain on an engagement is just simply not being ready. You've given me all the action items or you haven't and we get closer and closer to that date, that start date we agree to on the roe call and suddenly you tell me

5 days out that we can't test. We're not ready. We need you to wait three months, two months, one week. Doesn't matter. as a external pentest firm. The real hard part there is you've paid for my time. So now we have a choice to make. The choice is that I get to sit around and work on other stuff like presentations for things like this and you just simply pay me for that day that you lost or those five days or we penalize you a certain percentage of the contract because now I'm not doing work but I still have a salary that needs to be covered and I can't go to company o company B over here that's

scheduled three weeks out and say hey are you ready to start early because usually the answer is absolutely. So if you agree on a start date, you signed a contract, it was reviewed on the ROE call, everybody's on board with that, make sure that you are ready for that date. If you've hired a pentest firm and you have a c you have a pentester like I I can speak for for BHIS. If the customer is not ready for testing and we've had an ROE call 3 four weeks in advance and we find out that three days before the action items aren't due, the customer isn't ready and we can't move forward. The first question asked is you as the assigned

pentester, did you hammer them and badger them and annoy them enough on every possible bit of communication that escort? Because if you didn't, it's not the customer's fault they're not ready. It's your fault because it's your engagement to handle. But if you do all of those things and the customer is still not ready, I could say from our reports, you'll get an environmental preparedness for testing finding. Just anformational, but it's just a way to say you guys weren't ready and we lost two out of five days on your test. It doesn't look great to leadership to just say you weren't ready. So tens of thousands of dollars in these are the more antagonist antagonistic things that clients can do.

Um, and if you raise your hand, you're a client that hires pentest uh, firms. I guess you could pay attention to here, but you are great people. I love you. You keep sending money. And this is most not I've never seen somebody check every box on this list as a single customer, but I have seen this happen to myself and other pentesters I know directly. The first one is being adversarial or taunting your tester. That's a real quick way to get a very angry technical person to tear you apart in your environment and make you look really, really bad for the court. That is not our goal. We are not trying to make you look bad, but if you're a jerk, it's

kind of hard to be nice to you, right? Uh I've heard stories I I I've done a series of webcasts on the Black Hills uh webcast stuff uh called the illustrated pen tester, and there's there's four of those out there. all out there on YouTube. But one of the stories I've told there is there was a incident um with a pentester where we had an ROE call. The point of contact was there. They invited a few extra people on there including the IT manager who believed that his baby was beautiful and nothing was ever wrong with it. And you as a pentester on a pentest engagement should get zero concessions to get into the environment or anything like that. uh

come to find out that that person wasn't very liked in the organization because one of the passwords that was captured and cracked was his name and is a jerk. Um I'm I'm polishing over the actual language that was used there a little bit, but it was not a very nice first and last name of this person and and getting getting there. And uh take a guess which correct password appeared in clear text on that report. Um, threatening legal action really early in engagement is a big red flag. Second day into the test, you don't like how things are going as a customer, don't come to your pentest firm and say, "I'm going to sue you for all this

stuff." Because that's a way for us to just stop working. Point blanket period. You're threatening legal action. That's not going to be good. If you insult your pentester personally, professionally, based on anything about them, I've had uh pentesters that I know that they were accused of being too junior to be assigned to the engagement. Uh this particular pen tester, she's very bright and uh and very very capable in a lot of areas that I'm not very capable in some of those areas. She had DA on their environment within two hours and then she called them up and said, "Here's the problem. we've got this. It's a cr It's a critical issue. I'm not going to stop

testing, but I want to let you know that at this point, we can basically do anything in the environment. They didn't believe her. She had a draft write up of all the things, but they still did not believe her. Um, and they she wound up inviting other pentesters onto the engagement who had spare time, and it was a really bad report. Going dark, not being responsive, taking bad news badly is also very bad things you can do. Some examples from a tester driven pain point is just simply not knowing your tools. You don't know you don't know how much u you don't know how much damage your tool is doing. You don't know how to configure it properly. You don't have

to keep things out of scope. Those can be really bad because you wind up attacking things that you didn't need to. Adhering to that out of scope list that is provided at the beginning of engagement as well. A lot of these things like all this stuff at the top part of this is all about making sure you're staying within scope. If you're not paying attention to that, you're going to have a bad day. If your business type is a customer who If your customer is a business type that is a hospital or industrial control systems, food manufacturers are a big one as well. Anything that deals with the health, well-being, and safety of human beings, it's important that you drill

down on your customer. you're responsible for this, not your customer, of saying, "Is there anything specific in your environment that would do harm to life if I accidentally broke the thing?" That could be in hospitals. Uh, particularly, I've found in in hospitals, they're not terribly secure environments. They put a lot of trust in their vendors products. I've found instances where they're confict or worm infection on a operating room an operating suit imaging machine where they're doing live live imaging to help with live active surgeries in that operating suite. Uh the customer took it extremely well. Did everything stop testing, contacted them, isolated things, moved on. They handle they took bad news well. It's a good thing I know.

Um, but knowing that your customer's business type is an or is an organization that they don't have full control over the systems, uh, in that case it was a a couple million dollar machine that was vendor locked to a certain OS version, hadn't been patched in 20 years or whatever, and they can't just simply replace it. So, running DOSs exploits is always, it's weird that I have to put that out there, but I do have to put that up there. If you doSS a system and you know that you ran an exploit that says this could kill things that's bad. Biggest one though for testers is being too proud to ask for help. So if

you don't know something, you should be asking for help from your customer or your other testers in your organization. And the other biggest thing in there for testers for pain is that you're not reporting as you go. If you're doing a real bad job at doing your reports, if you're not leaving your report where I've got two young children that go to daycare, I get sick like every 11 days. Sometimes it means I have to step away for 24 hours and I have to be able to hand that report off to a certain to another tester that's on the bench and say, "Step in for six hours. Do these five tasks to keep the ball rolling so I

can do this." Inaccurate reports or lacking evidence in your reports is also a real bad thing. you. The report is the action. That is the thing you're getting paid for as a pentester. If your report is not actionable and they can't fix things, they can't follow along with your logic. They don't have good evidence to provide to their higherups and get uh mitigating controls or defenses in place and get the budget for those things. Then your report is a whole lot of heap of nothing for them, right? Uh I've had many tests where I get on the rules of engagement and I say, "What's your motivation for the test?" Most of the time it's just

compliance auditing. I need to do it because the board said so. Sometimes I get organizations where a point of contact says, "I've been trying to fix this thing for 5 years. They won't hire somebody to do this full-time and they won't give me budget to replace the system, but I could convince them because of compliance that we needed a pentest every 5 years or we would have some some fines coming our way. And I want you to tear us apart as best you can. Make it look bloody. make it look ugly. The funnest part about those conversations is usually a day later I'll get a side email or a phone call from somebody from that rules of

engagement call to say when you get started just take a look at like these four systems on this side of it. There's something wrong with them. I'm not going to tell you what, but it's definitely one of these four things. Could be all of those four things. Um but they want you to find things. organizations want you to want you to find the bad things most time at least the low-level people high up people want a clean report. Uh being responsive again and uh and also if you're getting onto a test type that you don't know really well it's okay to challenge yourself a little bit and expand but don't be the only tester on that engagement right and

you're trying to figure something out for the first time. that's just going to read to lead to a weak report, leave the customer feeling bad and they can do things. Um, so when things do go wrong, things like indicators of compromise, critical findings, or you you just got errors in your report, don't have an ego about it. I've had a bunch of reports where I run a debrief after the engagement is done and they say, "Hey, here's this additional context. We think this finding should be lower. We think this finding should be higher." which is a strange request. Uh but always if they provide context and evidence, don't have an ego about your finding about your

baby of a report because it is just a deliverable to make the organization better. That said, if you come to me on a test and you say I'd like this to be a low when it was a high, you I'm usually default answer is no because I I don't care about your personal opinion. and you hired me to do the job. But if you can provide some evidence to guide me, I will absolutely make adjustments because I'd have no problem there. And if I don't agree with you, I still toss it to five other testers to get their opinion. I It's not just me. Uh document, document, doc, document, know when to escalate, right? You don't have to be the person to say

no all the time. Have somebody in your back pocket. I know for us at at BHIS, if a customer is being contentious, if they're giving me a hard time, if they're being a jerk, I don't have to be the person to deal with that. I will just hand that over to our COO. You'll get a lovely phone call from him and he'll work out the issues or from a project manager, right? I think that's it. Let's see.

So, we do have a few minutes here for questions. Anybody? So, yeah, do that too. I'm on a uh government uh red team and so a lot of times we have reoccurring clients and recurring blue team that we interface with and it is a bit of an adversarial relationship. Um, so I'm curious other than setting clear expectations like you discussed, is there any other strategies you might suggest for us to work on that about constantly having recurring customers who are adversarial? Yeah, I could tell you my answer. Don't don't work with them. So after every engagement, one thing that we do is uh we obviously solicit feedback from our customers. How did things go from your

end? But we also ask the tester as well. So if the tester felt like you were a very poor customer to work with, you weren't you weren't ready, you were adversarial. Most of the time it it's if you're not ready, we can work with you. But if you were a real real jerk during the whole thing, uh, and you challenged our testers and talked down to them and all that, uh, we'll I wouldn't say it's on a blacklist, but we'll say maybe not next time, right? Uh, that might not be a choice you can make. Um, but a good thing that you can do there for the the specific tester involved in the engagement, don't make that that initial

point of contact person who's doing the work be the person that has to tell them no and back off. Have somebody they can go to that they can say this is out of my hands. You can talk to whoever. I'm just doing my work here. Yes. Uh, about the institutional bias for a clean report. So, I'm not doing pen testing, but I'm having to escalate about problems, namely with solutions. Can you speak to anything about helping uh senior leadership, executive leadership handle feedback internally, externally about a lack of clean cleanliness on a report? So, there's two parts for a clean report. Everybody likes a clean report. You shouldn't expect one. Uh when I say clean report on on on on this type of

deal, uh you should be happy when things are found, right? You hired somebody to do the work. They're trying to find the problems. Be happy that they're there. Make sure there's good evidence to support the severity of them and take that as something to do better. Even finding things, if you have a discoverable report in say a court case, showing that thing you took the steps to even find and identify the issues is a good positive thing. where it gets to a negative where you need a clean report. I I obviously can't say any names, but I I tested an organization that sold a product to uh school systems in various states and counties and they had a

policy where they were providing to the state and to these uh school boards or whatever their actual full pentest report and they had it written in that we will never have anything above a low in a pentest report find it. So they argued tooth and nail with me on the stuff and they said we can't sell the product if there's anything above O. And I said that's not my problem. I said but the we have to give this to our potential customers. They won't buy it if there's anything above that. I said you shouldn't be giving your report to your customers. I can give you a letter of attestation that will give you a very

vague write up that you had something done and a rough understanding. Um but I think we're well on time here. So if you got more questions for me, I'll be around for like an hour before I got a grunted report. Hey, let's give Chris a round of applause.