Pentest Pains

[applause] That was a mistake. Um, [laughter] I don't like to point out people's mistakes in public, but that was a huge mistake. All right. So, who was who came to my workshop last night? >> So, handful of people. Cool. Awesome. So, then you guys don't need this stuff up here, but there's some uh Black Hills stuff up here. Uh stickers and fun stuff. So, if you guys want to grab it afterwards, come up and grab it. Whatever you take, I don't have to carry out. So that's really really really helpful for me. Um, okay. So, uh, pentest pains, I know it's the end of the day. I think there's one more thing after this or something. So, I

appreciate you guys sticking around and coming to hang out for this for a little bit. Uh, we're going to talk a little bit about painful things that happen on pentest engagements, whose fault they are without hopefully being too judgmental uh, and aggressive about it. But uh I hate this slide. It's my least favorite slide of all slides in all existence. Uh but they make me do it. So uh my name is Chris Trainer. I am pentester at Black Hills Information Security. Uh I am the owner of Ridgeback Information Security as well, which is why there is a lovely little Rhdesian Ridgeback uh profile, I guess, up there. Uh that's my company. It exists solely so that I can build uh offensive

training classes and tools and stuff like that. So if you've got questions about stuff, we can talk afterwards. But if you were in my workshop last night, you got all the labs already for the two-day one. So, good job. You gota you got a bargain, I guess. Okay, so pentest pains. Uh why this is needed as a talk is is pretty simple. I try to keep my my conference talks very very simple. It's just because it's needed, right? Um either you are a pentester or you are a sock analyst or you're a project manager. If you're somebody that receives pentest reports, deals with pentesters, or if you're just somebody that approves money to uh to spend on

these types of offensive engagements, this type of information is really valuable. Even if you're not ever directly involved in a pentest, it's still valuable to know this because it goes beyond that first degree of separation of an actual test. Uh just real who's a who's a pentester, offensivey person in here? Couple hands. Any defensive people? admins management style. Okay. Uh I just like to try to get a little bit of a gauge of how many people I'm going to be insulting for the next 20-ish minutes on each side of it just so that I know uh who to hide from later on. Uh so painful engagements have a lot of consequences, right? Uh first and

foremost, it's a less secure environment. That's the opposite of what we're going for. If you have a painful pen test and not everything gets done that should get done, then you're going to wind up in a situation where your environment's less secure. And that's obviously the loss uh of the main point. Loss of revenue being the big one for if you're a pentest firm, you're going to not win out on any money. If you have a painful engagement, you may not get that return customer next time. And obviously, that is just an easy thing. Uh, one that people don't think about very often and I could tell you if you ever do work with Black Hills Infosc, we

welcome your feedback as a customer for the tester that you got. Keep it constructive. Obviously, if I'm your tester, it's always good five stars, thumbs up type of situations. Uh, but after an engagement, the tester, I know for us, we uh if we have a particularly painful engagement with a customer, we'll state that to our sales team next time, right? you're not going to be like blacklisted or anything like that where you won't do work. But if you were a little bit difficult, I know there have been some engagements where there was just one person out of four people for the company where they were just aggressively difficult the entire time and we've had to take the stance that if

we want to do work with us again, that gym cannot be involved and we're sorry to say that, but that's the reputational stuff that we're dealing with here. Uh litigation is something that unfortunately does happen in pentesting. you're conducting an activity that in the public space is a felony, right? So, some people don't understand that when they sign that little piece of paper, giving you the ability to do these malicious acts that they are also signing away a lot of their litigating rights. Uh, but it does sometimes uh lead to termination of of relationships. Uh, sometimes terminating employees as well. Oh, and the very last thing. I usually am able to do this little my

little laser pointer, but these are not projectors. They're back lit, so it doesn't work at all. Um, I'm going to spend this next little bit of time complaining, uh, but in the best possible way, hopefully. But everything that I'm pulling up here from both sides, this is just an aggregation over years and years from many testers and and myself, personal experience. Most engagements, nothing bad ever happens, right? The relationship is great. Point of contacts are great on both sides and things work very well. So don't if you've never been a pen tester involved in it. These things are not the norm. These are just stuff that stands out. So the hardest thing has anybody had an

outdoor job before where you're actually working in the sunlight and the heat and it sucks. Yeah. >> Does dog walking count? >> Dog walking counts. Counts because you're literally dealing with crap. >> I had that job back in 2019. Yeah. So it can any outdoor job is by definition harder than the jobs we do sitting behind a desk. But the hardest thing about sitting behind the desk is managing communication and expectation management with people on both sides. All of these problems can be solved by you just managing expectations correctly and communicating very well. uh preconceptions of the type of offensive engagement or the type of activity for security that you're doing where those conceptions are in incorrect. Right?

There is a a fundamental difference in what you're trying to accomplish with a red team than you are with a p with a pentest and you are with a purple team and you are with a social engineering engagement. There are fundamental goals that are different from that. So, for example, if you're doing a pentest and I'm your pentester, I'm going to ask you to allow list a lot of stuff through your environment because I'm not interested in getting past your defenses in a pentest. I'm interested in actually finding out if those systems behind them are secure. If you want me to be sneaky, you could pay for me to be there for a much longer time, and that's called a

red team. I will take a lot longer to burn through your environment, do some evasive stuff, and be less noticeable. But for a pen test, I'm going to be sprinting through the halls of your network, banging pots and pans as loud as I can because typically I have about five days to be there and then I'm gone on to the next thing. Uh my wife is a she's worked for the Navy for her whole career as a civilian and she tells me all the time that I tell very long and convoluted stories. So she is a very big advocate of bottom line upfront bluff which is hugely impactful inside of uh military structure. So bottom line up front, you

can avoid most of the stuff that I'm about to say by having a good rules of engagement call where you set out these scenarios and these plans. Establish good points of contact. Cover PTO. I've had a a point of contact in the middle of a test failed to mention that he was going to be traveling back and forth from the Midwest to New Zealand during the 5-day test. Things went wrong. He didn't pick up the phone and we wound up pausing testing for two and a half days until he decided to return the calls. Uh it's not great. Uh, always find a backup point of contact to reach out to. If you're ever going through a rules of

engagement call with a with a pentest firm, always establish what the primary motivation for the test is. You don't have to tell me that you got breached from this system with this thing like 6 months ago and that's the reason that you have to be having this pen test. But if there is particular concern, it's always good to get that question out of the way weeks before this test ever starts. I've had some customers on calls where they'll say, "Well, we're just a little bit concerned about some stuff. This is just a run-of-the-mill thing." Like, "Okay, I'll note that down. That's no problem at all. This is just a run-of-the-mill annual type of pen test." And then there's five people on

that side of the call. Afterwards, I'll get an email from one of those five people saying, "Here's a list of four or five very specific areas of our systems that I am concerned about, but I can't get the other four people to spend the money to fix it." So if you just want to look over there a little bit more and kind of rip it to shreds in the report, it's much more ammunition for people to get things fixed. Big key one for painful engagement avoidance is is defining what is in scope, what's out of scope, what can I touch, what can't I touch. And there's a few gradients in there of what's out of

scope. It could be out of scope, you can scan it. It could be out of scope, if you touch it, you'll be fired. And it could be out of scope as in well if you think something is messy over there just ask permission first and I'll probably give it to you but I need to know ahead of time. So out of scope is not a whole who wholesale scenario. We'll talk about emergency contact plans, things like that as we go through things. So there's two sides to the rest of this talk. One is the client-driven pain. And if you're a client side of things, I do really appreciate the business that you drive towards us as

pentesters, right? I get to break the law for a living and you don't send me to jail, which is super fun. Uh, probably the most beneficial part of it is that you pay for the daycare. That's super expensive for my kids. Uh, so I do appreciate clients. I don't want you to look at these next two slides and get very upset at me. I'm going to give equal time to the opposition of the other side of the table here. So, one of the client-driven pains that is that can really screw things up on an engagement top of the list has to deal with that scope situation. How you provide me the scope is how I interact with your

targets. If you provide me a single list where you next to a single IP out of 6,000 say, "Oh, by the way, this one shouldn't be touched in one single long list," that's going to be real messy and hard for me to just pull out of this giant list. So always separate what's in scope and then what is out of scope specifically as a list. If you're giving me cider notations, who's familiar with cider notations? Networky people, you're very familiar with it. Okay. So if I gave you a 1 192.168.56.724 as the scope, what would you interpret that as 7/24? If I put that in, this is in scope. Here you go, pentester. you'd be possibly understandably

confused. Is this mean that it's just this 7 system on that subnet? Does this mean that it's all the SL24s and you just screwed up? Well, you won't be surprised to hear that that's a confusion on both sides of the table. So, client driven. Make sure you're very specific. Give ranges of IPs. Try to avoid cider notation unless you actually say.024 or something like that where it's very clear. Another painful part about that is if I plug in your scope to some of my tools, one tool is going to interpret that cider notation as a single system. Another tool is going to interpret it as all 255 possible IPs in that subnet. I can't know for sure every single

tool's way of interpreting that. So clarity is key there. Anybody work for a university or a hospital? Somebody who's had the internet in their environment since the internet became internet. those organizations tend to not uh comply with RFC1 1918. Does anybody know what RFC1918 is? Yeah. It's the thing that tells you these batches of IP addresses should only be used on internal environments. The reason why that's important is if you wind up not adhering to that and you wind up assigning things on your internal network, what are public IP addresses out on the internet, if your routing is screwy, my attacks could accidentally get routed outside of your network and start attacking real systems in the

world. There is a story about that that I don't know all the details and I don't know how much I can tell but there was a situation in a test where the customer did not adhere to RFC1 1918 and an attack accidentally routed outside their network to a foreign adversarial nation state. Um in those situations you have to start talking to homeland security and a whole bunch of other fun conversations. Um, another big one for for client-driven pain is if the date comes and you're not ready for me as a pentest firm, guess what happens? I still get paid. You just don't get tested, right? If you are on if you're on the client side of this thing and you

are having a pentest come up, you should have heard from the person who's going to be attacking your system directly. If you're with a pentest firm, I'm not going to trash talk too many of them, right? But if you're the pentest firm and you've paid x thousands of dollars to try to have a pentest and the day arrives and you've never had the pentest person themsself reach out and say, "Hey, it's two weeks away. Hey, it's 10 days away. Hey, it's 5 days away. Are we ready?" Then that firm is probably not a great place. They're probably just a pentest puppy mill. They're just pumping these things out for the dollars and you're not getting a great value. If you want

that as a checklist test, stay with those people. That's great. But I know on my side with BHIS, if a test isn't ready to start the day of, the first question for our organization is, "Chris, did you follow up with them enough?" Because if the customer isn't ready, but I didn't follow up leading into that engagement to make sure they were ready, that's my fault. But if they're not responsive, there's nothing more I can do. So what you probably gather as an easy thing here is don't be adversarial. We've had engagements in the past where uh there was an IT manager on the customer side of the of the call who leading up to it throughout the

engagement. That IT manager who's an IT manager? Like you're in charge of everything. Nobody in here. Good. So this won't offend anybody. All right. What you put together might be your special little baby. Nothing's wrong with it. It's beautiful. Um, but we're there to break it. And if we get on the call leading into the engagement and your response and your your attitude towards us is we're the adversaries to you instead of people you've hired to help you be better as an organization, things are going to go south right away. Uh, there was an instance where, let's say the name was Bob, IT manager Bob, he was very upset that we would have the audacity to attack his system. He's not

going to have any allowances, nothing, nothing to make sure that the engagement went well. He said, "If you want to break it, you've got to earn it. You've got to do it completely on your own. Not going to help you one little bit." Um, you'd be probably not surprised to find out that we actually got a number of hashes on that engagement, cracked those hashes because they're fairly weak. And more than one of those passwords include trying to gauge the audience and keep it uh to a to a swear jar minimum. Uh, expletive Bob exclamation point as the password. So, yeah. Uh, taking bad news badly is a really is a really terrible thing that you can do

as a client, right? Uh, if I'm telling you things are bad, I'm trying to help you. If you if I tell you something's bad and then you get angry at me for your bad environment, that's not going to help anybody. Uh, big one here as well is having the wrong motivation. If your motivation is to get a clean pentest report out of me and when I deliver it to you, you're upset that there are highs and criticals on it and you want to argue every little piece because you say you can't sell your product unless you have nothing but lows on the report. That's going to be a big hard no for me. I'm not going to

negotiate on what we say is bad because of your d your directive. There's a lot more on there, but we don't have a whole lot of time. So tester driven pains on here. Knowing your tools as a tester, knowing your tools as an offensive operator is your job, right? If you don't know what your tools are doing, then you're not doing your job. You're just swinging hammers around with a rope at the end at the end of a rope and you're just hoping not to hit anybody in the face. Uh, that's a really weird metaphor, I know, but it's what you get if you have testers that are just firing off tools they found on the internet without any

te any figuring out is what they actually do. Adhering to that out of scope list that you actually agree to before the engagement is a big one. I've been on the other side of this where I have I have tested something and they did specify out of scope. Uh does anybody use Nessus vulnerability scanner? If you're ever configuring Nessus for an engagement, know that Nessus doesn't give two hoots what you put in your IP tables for your firewalls. It doesn't care. It ignores that rule completely. It will attack anything in there. on one engagement I forgot to set the Nessus exclude file for those uh for that one particular system and I actually did hit it and

that was a a boo boo on my part so I said sorry uh [clears throat] obviously don't run any sort of denial of service exploits on engagements those tend to go badly when systems go offline and customers get angry at you right [snorts] um have have some regard for the type of engagement that you're on who your customer is anybody work. I know I asked universities, does anybody work for hospitals or food manufacturing plants or industrial control systems? There are certain things that I would do on a regular company that I would not dare to try in those environments because when those things go down and go offline, people's lives can be affected, right? Balances in chlorine and water

like we've heard in this in news reports over the last several years of ICS systems can go really badly for people as well as food or particularly hospitals. If human life is involved, you need to have that explicitly called out in your out of scope. Those systems should not be touched reporting. Anytime I tell somebody that I'm a pentester, they must think that it's the fun awesome job. All I do is break things and do stuff. I spend at least half my time, if not more, in Word. I don't spend it in the environment because as I'm doing stuff, I need to report and document everything I'm doing. There's a couple good reasons for that. One is you want to make sure you

don't miss any evidence when the engagement's over and you have to polish the report and deliver it and you forgot that one screenshot. But also, if I get sick, like I'm probably getting right now or getting over because my voice is going, I need to be able to hand that off to another tester to pick up where I left off without a misstep on the engagement. So, if I go down, we can't go back to the customer on day three of a five-day engagement that they've prepped for and say, "Sorry, Chris is out." But another tester can't pick it up because we don't know where the hell Chris was going with something because he didn't write it down.

Every report that you write, you got to make sure it's actionable. Write it to the audience that you're that you're doing this work for. Does anybody do web app uh development? If you're doing a pen test for a web app, know there are certain methods and ways that development happens for software. Knowing that world will help you write a better uh finding so that they can actually address the issue and give them the ammunition they they need. Uh missing deliverable due dates. That's a little bit uh a little bit lower on the things. Uh, a big one is is that last one for being ethical, being unethical or immoral, and knowing where to draw the line. I've done fishing

engagements. I've done social engineering engagements. if I was on the receiving end of it and the organization thought that it was a good idea to try to hurry and stress me out and get me to turn over information because they were say saying my child was hurt at school and is at the hospital or something like that or my wife had an accident or something like that and they're trying to just like get information with these horrible horrible ways to to spin a story to get you going which is what real attackers will do. Uh but if you are an ethical person don't do that. If you if you do that to me, if you're in

this room, you do that to me, I will find you. You won't have to send me a report after that. I I I'm gonna take that extremely personally. Any indicators of compromise on engagements? It's like, was that Vanna White, right? [laughter] I feel like you should be up over there just sliding that across, right? [sighs] Uh so when when things go Did I miss a uh Yeah. Oh, last one there for tester driven pain. I've been in in the situation where um I am taking on a new type of test. I've never done it before. We all do those things. We try to learn something. Eventually, it comes to the point where you've got to actually do the work

yourself. Know that it's okay to challenge yourself on those. Don't do it if you're not in an environment where you have a team or other people that you can ask questions to. I've done pentests, all types, for years and years now. If I'm doing a test I haven't done in six months, chances are I've forgotten some of the commands or things to do. So, I have a bank of other 30 or 40 testers that I'll throw the dumbest the dumbest junior questions at and with no judgment, they answer thankfully and I try to return that favor. So make sure that if you're challenging yourself on a new type of engagement that you have uh that depth and you have no ego to ask

for the help as well uh whenever you need it. So when things go wrong like indicators of compromise are found inside of an environment. I told a story last night where I found a uh a malicious infected device on a hospital network. Uh it turned out that after hunting it down, it was actually the the imaging device, the camera that was used inside of a surgical suite where they save lives, but that was trying to infect all the other devices on that subnet with something that has been patched and outdated for 10 or 12 years at this point. Um if you come across an indicator of compromise that's not your actions, stop what you're doing

immediately. You go back to that emergency scenario plan that you laid out in a rules of engagement call. Pick up the phone. Don't send an email. Don't send a text. Pick up the phone. Call the customer. Let them know what's happening. Draft a write up of what was happening, where it was found. Doesn't have to be perfect. Put that somewhere where they can see it. If they didn't answer the phone, call every 10 to 15 minutes for an hour. And if they still don't answer the phone, get your supervisor to actually call their supervisor and start escalating it up because there's no time to waste in those situations. Uh, same deal if you find critical findings

or if you're a customer and you find something that's wrong in a report. If you find something that's wrong in my report, I mischaracterized something, I misjudged a scenario, I'm completely fine and open with that. I've learned over years and years of doing this stuff that I I can't have an ego when I deliver something, right? I may push back and say, "I think you're wrong on this, and here's the reasons, but I will take no offense if you question my report and you ask ask for more details as to why, which is why it's down there. Don't let ego get into the way." And remember that you're all kind of on the same team here.

Always document. Document document your report. Document your communications with your customer. Remember, I said that if the customer wasn't ready and the first question back is, "Chris, did you follow up with them enough?" Phone calls can't be traced. My follow-ups are in emails. Those are backed up. They are CCD with the project manager on it on our end. They have three other people CCD on it. It goes into the project tracker in there as well. So, if I step away, I die, something terrible happens or I something good happens and I win the lottery. They know that communication happened. It's not just buried in my text messages or my emails. Know when to escalate. When things do go

wrong, there should be points of contact escalate on their side. There should be points of contact escalate on your side. As a tester, as an offensive operator, the last thing that you should be doing is putting your position, putting yourself in a position where you are the bad guy. You are you should be the objective person, right? If a customer gets adversarial, if they've threatened legal action early, law firms love to do this because they're lawyers and that's all they think about. If they start threatening things like that, that's the immediate point where you just stop what you're doing and put it in somebody else's hand to resolve the situation because there's no no way more waste of

time than having a tester try to argue with a lawyer. It's not going to work out well, right? Nobody's going to move forward and it's going to be bad. So, hand that off to somebody else so you do not have to be the bad guy in that situation. That's it. That was tight. That was That's close. It's very close. At one minute, does anybody have any questions? I have a ton of stories around every single one of those, but I have to keep it to 20 or 25 minutes. Um, so yeah, any questions about that? Everybody knows how to run a perfectly painless engagement. That wonderful little QR code thing is something that John Strand, uh, Black

Hills founder, um, started throwing up at the end of slides and they're kind enough to send me around to a bunch of places all over the world to do these types of things. If you want some free labs, that QR code or that tiny URL will take you there where you can try out a bunch of cool environments. These are different for those that were in my workshop last night. These are different than what you got last night. So, any questions? Any questions about dogs? >> Rhodesianback in the world. >> Why is the Rhodesian Ridgeback the best dog in the world? Uh, very simple. It was bred to hunt and kill lions in South Africa. Uh, it can survive in hot and

cold climates. It can run with a horse for up to 10 miles. It's amazing with children and very, very good at protecting you. >> My children are lions. >> Your children are lions. It's probably not going to work out well. Yeah. I don't know. Yeah.

>> How hard is it? when they have little to no experience. >> So, that was a that was a good question. It's if you have a handful of certifications but you haven't really done pentesting before, uh what's a good way to get into it? Um I can talk about that for probably an hour and lose my voice. Uh the quick question the quick answer is find some place where you have the I guess latitude or influence where you can try out new things. I spent I got out of school and found out very very quickly at that time way back uh that cyber security nobody really gave a damn about it. Um they didn't care in

organizations, right? Smartphones hadn't been invented yet. People weren't carrying their laptops home with them. and laptops weren't really a big thing then. So, um, I spent a decade not doing security and trying to convince all the places I went to to let me do security and they all shrugged their shoulders at it. It took going to I went to work with Equifax two weeks after the public breach announcement because they couldn't say no to security. They were big enough to weather the storm of a breach, a hugely public breach, which in my opinion is not the most impactful breach that's happened to our country or any organization. Um, but I could talk about that another time. Uh, but what

they could do is they had a pile of cash and they said security is very important. We're in the public spotlight. We need to do this job. We're one of the three big credit bureaus. We're not going anywhere. So going there, I took a non-security job on a on a software dev team uh doing QA automation stuff and I made it very very explicitly known as I walked in the door that if any security related concerns come up I'd be more than happy to do that for the team and within two months that became my thing. They said, "Hey, you wanted to do this." And I said, "Well, I'll do that, but if you get me

good training and actually have my back, so I'll agree to that extra work on these conditions." And I was able to strongarm that relationship as nicely as possible for about half a decade. And I tried out probably about four or five different types of security roles uh before I decided I hate defending things and I like breaking things. So yeah, but uh if you go to uh Ridgeback Infosc on there is just all of my stuff. There's recordings of other talks and other stuff that I've done on there. And if you go all the way down on the bottom, there's probably three or four um interviews where I talked a little bit more about how I got into security and

and the the mindset and the approach that I did. I can't answer everybody's scenarios, but it's how I did it. Anybody have any other questions? No, this is a different shirt from last night. I have four of these. I know that was the question burning in everybody's mind. Uh, come up here, grab some of this stuff up here. Take it home with you. If you got questions about Black Hills, got questions about offensive tooling and training classes, come talk to me. I'll talk till my voice goes out or 30 minutes runs up. [applause]