Counterintuitive Outcomes In OffSec Consulting - Simon Chapman

So um I I'm framing this talk as a as a a about offset consult. It doesn't We have a microphone failure. Hold on. Let's do this one.

Let's try that. Okay. Well, good. Um, but it it kind of applies to any consulting really. Just as uh just out of interest, is anybody here have or want a career in uh offensive security, pentesting, red teaming, that kind of stuff? Quick show of hands. Okay. Anybody um working around that kind of field? So, working with pentesters, implementing recommendations. Okay. A little bit more. Okay. So, I guess you all know where I'm coming from. Um just a quick introduction. That's me. I took a a gamble on which side of the uh screen I would be. I got it wrong. I'm on that side. But okay, guys. I I run a company called Comack and I help pentest teams uh with their

leadership and development issues. So, I guess we all start our um offensive security careers or indeed any career with a with a great deal of enthusiasm and optimism. And that's and that's a great thing. And we all like to think we're making a difference and we're trying to help our clients. And that's certainly what I thought. And I I still do think that to some extent, although it took me a long time to learn some of the things that we're going to talk about today, which is either because these things are very difficult to learn or I I'm a slow learner. Um maybe a combination of those two things. In reality, um consulting isn't offset

isn't just about finding vulnerabilities and exploiting them and so forth. There's a lot more about navigating the relationship with the client and this is the where the counterintuitive bit comes in because when you start off you think it's a very tech heavy thing which of course it is but there's this situation where you have to navigate that client relationship preserve your own integrity and reputation and also do what the client wants. And that's the bit that's really quite hard. And if you put alongside that the fact that not everybody takes security as seriously as maybe we do in this room. Um, and you you realize you have to work a lot harder to be heard than than you think

you might. So, let's get straight in with the spoiler. I've talked in in in the talk synopsis about uh misconceptions and the biggest misconceptions. And these are the biggest misconceptions that I've picked up over many years. And one of them, the best one is that our work will speak for itself and the clients will automatically understand and and value it. And in reality, that isn't the case. um it's quite hard to be understood and what seems obvious to a seasoned tester is actually confusing or or difficult um for for a client and therefore clients don't always see the value of that work. The other thing is that we think that clients are going to share our

priorities and that everything that we do uh every test we do that finds issues the client's going to be grateful about that. Everybody wants to do the best possible depth of testing and again my experience over the years has been that is not always the case and in many cases clients actually don't want to see certain things. And the final biggest misperception is that technical correctness is in fact the the equivalent to being right and it always wins the argument but in fact it doesn't because right is a is a very subjective thing. um and a client doesn't automatically embrace it. Um the idea of right to one person is not right to another. We have considerations about

budget, time scales, business, um conditions, uh and even internal politics always can influence what the perception of right actually is. Perhaps a useful thing and this is a theme I've seen in a few talks today. It's been repeated. uh we we're we're living in a a really interesting kind of potentially a golden age of uh a generational change in technology with with AI kind of dominating the landscape. Um and there's never been a better time to reassess how important is to actually know what you're talking about. There's a lot of tooling around these days that will actually do a lot of work for you. And if you're not in charge of that, if you don't actually

understand what you're doing, then there's a very strong chance that you you will be automated out of it. Um there's a lot of uh really interesting and and compelling developments with in security tooling. Um but we must avoid the possibility of becoming mere operators in that scenario and remain in control of what we're doing and being knowledgeable of what's happening knowing when we run a tool knowing what the tools actually doing rather than just running a tool and expecting a certain result. And of course that's not a new thing but it's actually it's being exacerbated by the increased sophistication of uh AI based tooling. So here's another a counterintuitive thing relationships with clients. Now clients are um

obviously our they they pay our they pay our bills. They keep the lights on. It's a very important relationship in a commercial setting. But it's not a friend relationship. Uh now some people will say I have clients I regard as friends and you know okay maybe they do and people have client relationships which are very good and very and and very productive and that's fine too but it's not a friend relationship. There needs to be some professional distance between you and your client. Sometimes you will have to tell your client things you wouldn't want to. It's not a friend conversation and there's money involved so it's automatically not a friend conversation. Professional distance is important. Really, the best client is

one that challenges you anyway um and forces you to defend your work. So, let's talk briefly about red flags in that client relationship. Again, remember these are things which seemed initially straightforward, but on uh further inspection and in retrospect were actually red flags and actually uh signs that I was about to compromise my own integrity or say something that's going to get me into trouble. Um here's some here's here's the act activation phrases for the for the the red flag in the bottom of your mind. Now, some of you will automatically recognize these kinds of conversations that you will have whether you're in offensive security or any other kind of consulting. When somebody says, "I just want your opinion on

something." Um, that's all of these things you can respond to really quickly. You can knee-jerk react to them and say, "Yeah, yeah, okay. Well, here's my opinion." But opinions have a funny way of becoming fact or becoming something. That's what he said. That's what she said. Um so whenever that happens I always think we should take a little bit of time to think about it. Another one which is particularly frustrating for anyone who is working in pentesting or offsec is where you get compared to what the last company found and that's something which is very hard to respond to and when you're getting into that conversation you have to think why am I being asked this it might be

that the client is trying to frame this conversation or frame this engagement in a way that undermines your your integrity or your independence alongside the classic I'm not going to hold you to anything but whatever uh if there's any sales guys in the room or people involved in in commercial aspects in the client relationship, you'll know that when a client says, "I'm not going to hold you to this, but how much would it be?" "You know that that's basically exactly uh what the figure what the perception is going to be from there on about about pricing." And it's the same elsewhere. Um other other things which are a little bit frustrating is where the client is trying to frame the the

engagement as um a black and white pass or fail pentesting activity which is you know a misconception thing. Uh in activating in in answering these questions we run the risk it turns out of compromising ourselves. So there are things that are very difficult for us to defend professionally and they are things that all the other red flags are indicators that we may be about to be asked to do. Um has anybody ever had a client ask or have you ever asked a pentester to remove a finding from a report because it was somehow inconvenient? I'm seeing some nods here and there. Yeah, we don't talk about it in public. Well, we're going to talk about it today, but um it does happen.

It's something where uh our client might be and it's not a great client is asking you to do this but um it's uh an indication that they're going to use your reput then your reputation is your responsibility. Uh and if you do these things it's you know it more fool you. So looking the other way is another thing. You don't need to look at that that's been fixed or that's not an issue you need need to find about but it's in the scope but I don't want you to test it. These actions are indefensible because later on we have to answer for our own work and if we do these things um how do we um how do we defend that?

How the client told me to do it. Who the hell's in charge of this engagement? You all the client really you should be in charge of what you're

doing. There's also the unanswerable aspect. Um I do like it when I get asked questions um about things I can't answer. How secure are we? I mean that and that's is completely unanswerable. And that's I mean okay I know there are certain things where there may be metrics involved. There may be actual data, hard data involved where you can say we have improved, we've fixed these vulnerabilities, you know, whatever else. And and I get that that's okay. But the open question, how secure are we um is is completely unanswerable. And really not the job of a pentest anyway. The job of a pentest is to is to locate, exploit, and offer remediation. It tells telling the client something they may

they don't know or validating some assumptions. It's certainly not to say you are secure or you are not secure. Um, that's a yet another pit, yet another uh uh rabbit hole. How do we compare? How is how likely is it we'll be hacked? If we implement this product, would it help us? Um, again, really, really thin ice there. I'm not going to recommend a particular customer, uh, implement a particular product that you speak to those guys yourself. Do your own market research. Yes, I have an opinion. Maybe it's an maybe it's a useful opinion, maybe it's not, but it's just an opinion. Um, if the if the product doesn't actually work, then all of a sudden it's going to be my

fault. Um, another great example, something I've been involved in very recently actually, is where a client comes back and said, "A researcher found this issue. You didn't find it. Uh, your team did not find this." And it's really obvious. Well, when I looked at it, it wasn't obvious at all. It was extremely I mean it's obvious when it when it was there and a researcher would spend six months finding it during the two weeks we were working on it. It wasn't obvious at all. So again those are the sort of questions you can't answer. And the real the trick here of course is to not engage in answering these questions as far as it's possible. I see testers all

the time being bounced back and forth in this these these arguments stemming from these unanswerable questions. Um, and the best the winning strategy is just not to engage. I'm not saying we should ignore our clients, but you get my point. Here's another thing which is counterintuitive and that is that um being helpful isn't really helpful for a number of reasons. uh and the and the two the two things that a lot of times and I used to do this all the time were I think has been helpful for the client was overd delivering and working more time than was outside of you know outside of the test window. The overd delivering thing absolutely bites you

back in in in in the backside later on because um sometimes the client doesn't actually thank you for finding more things for them to fix. They see that all the issues as problems for them to resolve. If you hadn't told them, they wouldn't have to fix it. That's a even that is a counterintuitive thing, but a lot of clients think like that. A lot of people think like that. You've given me you've given me a problem to fix. We have we've just the problem's already there, but now you know about it. Um working outside the test window has um more than once caused issues where um we weren't actually aware that the client was monitoring what we were

doing. And turns out that somebody working offshore at 3:00 a.m. caused somebody in New York to be pulled out of bed because of some um stock related incident we weren't aware of. We didn't think that that was a problem. We didn't bother to check. We just thought we were trying to be helpful, but we weren't being helpful at

all. So I begin this slide with a quote from the from the great Clint Eastwood. Um, opinions are like Everybody has one. Uh, and that's true, but um really we we get called to to give our opinion and and to to offer that advice all the time. But when we offer it, we have to remember that our outlook is different. Our perspective is different. We also have to remember and this is something which took me a long time to learn is that when you give your opinion or your advice it's some sometimes taken extremely literally. So when you for example in a report you provide an example of an exploit which might include a little bit of a bit of

JavaScript or something showing look at this I can pop up an alert box there you go I've proven the point there's an injection point there. So the client whitel lists that specific example in the report that you gave when then they say well it's been fixed and you say well no that's that was just a generic example you need to filter things more fundamentally than that that's not what you said in the report okay those sort of things happen quite a lot so you can alternate between being ignored and not taken seriously at all through to being taken very seriously and absolutely literally so it's important to us to differentiate facts from opinions be very clear work out whether something is

we're giving an example or we're giving the very specific thing that needs to be done and talk about caveats in a minute. But draft recommendations and caveats to to uh robustly defend our work. All these things took me a long time to discover. Okay, this is not mandatory and it's not really audience participation. So this completely false falsely titled. However, um just maybe just a show of hands, here are genuine examples of things that happen quite frequently. Um to anyone, any of you involved in this kind of uh work, you'll probably be familiar with these types of scenarios. Um okay, a quick show of hands. Anybody think it's okay if the client says we need more money for a

security tooling and uh ultimately we need you to uh raise the severity. No one does that. Okay, just me. All right, fine. Um okay, of course it's not okay. How about it's okay to to a pen test on a client's competitor as long as the client said it's okay. No. Okay, that's pretty that's pretty straightforward. The system owner has to give the permission. Okay, we got that marker finding is fixed if the report in the report if the client confirms it's resolved. No, you got to work it out for yourself. You're going to report what you see. If you if you're reporting opinion other than fact, you're empty ice. What about removing a finding from

the final report once you've confirmed it's resolved? So, you're removing all evidence of it. No, not a good idea either. If you remove that, uh, then basically you're removing the indication there was ever even an issue to begin with. So that's not an accurate trail. And the last but not least, client rings up, says, "Sorry, I'm on the road. Uh, you can start testing. It's fine. Just take this message as authorization to test." No, I wouldn't either. To test what? You know, I didn't say you could test that. I said the stuff we talked Confusion. Confusion. Okay, so I'm going to pick up the pace now because time is slipping by. How do we survive this? Well, we

survived this by having a toolkit. Uh I talk about AC/DC all the time. People who worked with me in the past, although this is one of the things I bang on about. Actionability, correctness, defensibility, and clarity. These are the these are the things our work has to exhibit through every finding, every interaction, uh making sure that something can be done, it's correctly identified, we can give reasons why we've made that recommendation. And of course, the whole thing is clear. If you observe those characteristics in your work, you won't go too far wrong. Also, the scrutiny aspect as offensive security people, we are under scrutiny. We do have to defend our work. And you have to be prepared at any time to

answer the question, those those questions. What were you doing? Why were you doing it? How do you explain what you got? And if you're running random tooling in someone's environment without knowing what's happening, you will not be able to answer at least one of those questions. Let's talk briefly about the art of the caveat. Caveats are important because they influence the test. Well, they they they are they describe factors that influence the test in some way. Uh and they're not an influence. They're not an excuse for adequate testing. But what they do is they make sure that in the future when nobody remembers that the test targets were unavailable on time, credit didn't work, the test start was

delayed, your contact went on holiday for two weeks without telling you. All these things get lost in time. But if the caveat, if the report specifies that they were caveat caveats for the test, at least there's some context that can be applied. So this is um this is if you take nothing away from this talk um you should uh take away just this um and that is that we all want to be taken seriously of course but and there are many things that clients don't take seriously but we should not be one of those things and being taken seriously begins with uh self-awareness and how the client perceives you and that difference between how you're perceived

and how you think you're being perceived. And this is the thing that took me the longest time to learn and has been the most valuable lesson over the last quarter of a century. I sometimes see myself and we see ourselves as all of those things there on the left, confident, authorative, telling it like it is, technically strong, defending your point when you know I'm right. All great, all great points. I'm sure you'll agree. However, in in the wrong circumstances, it comes across rather differently to the client. And it's very you only know this is happening when you start when you can feel the frustration rising. But you have to work on this sort of thing. Um,

this disconnection between perception and reality is a cause of stress and not being listened to is not easy to live with. This isn't really a self-help talk, so I'll I'll I'll uh I'll move on. But that's a very valuable lesson. Took me a while to learn that. Um also, it's not all about us. We have to remember that we are part of a bigger picture. We're we're only there to help in a small part of what the client's trying to do. However, our work and our reputation is is our responsibility. A bad work lives forever. A report goes out there with your name on the front. It's all it's out there forever. All this takes

practice. Um, very quick deviation. We are not here to save the world. We're not vigilantes. We're not exempt from the law. People have said this in other talks as well. And I would say that I think I believe that the there's a narrative that somehow we are here to we are kind of saviors in some way. And I think that's that's not a great that's not very useful in offensive security because it means if we believe that we believe we're here doing something for you know a higher calling of some kind and we stop listening and then if you stop listening we start coming across in the way I've just described and then we get we we get all the stress associated

with that. We always talked about how being right is subjective. So I'll breeze through that one. So let's let's reframe some of the misconceptions I started off with. Ultimately integrity is part of our own uh our own reputation, our own defensibility. Our work has to be defensible. We have to listen to what the client's saying. And being right is something which is in the eye of the beholder. Though if we understand those misconceptions, all of a sudden our life our jobs get a lot easier. So, some positive closing thoughts. There's a lot of negativity in what I've said so far. I actually love our industry. I think it's great. I think we're living in a great time. As I said

at the beginning, um it's a great way to lead to work at the leading edge of technology and thinking. There's we're in the we're at the edge of a generational shift right now. The next few years are going to be great. I have no idea what they're going to be, but they are going to be great. Um, and that is all I have to say about that other than if you want to follow me on LinkedIn, please do so. Thanks, [Applause] guys. Well, it's nearly the end. Did you finally get it and you managed to clap at the right time? Well done. Uh, any questions? No. Honestly, come on. It's just Yes. Good. Right. Get the steps in.

Come on. This is your opportunity to learn from people. Ask interesting questions. You've just had an interesting talk. What's wrong with you? Just like my students. They always sit there and don't ask me anything anyway. Uh very interesting. I'm a consultant, but I'm left left shift. So, I don't want penetration tested. I should have done all the good work beforehand. Um the a word I keep on using is accountability. So where does accountability lie in terms of what we as security professionals do and where the business lies and the the fuzzy area between it it is a fuzzy area. I would say that personally we're all accountable. Like I mentioned there briefly, our own reputation, our own um is is our own

responsibility. Our own integrity is our own responsibility. It's our name on the front. Yes, you know, maybe a corporate brand on there as well. Employers brand is on there, but it's your name. It's our name on the front. Um the the I mean there's there's a whole discussion about in certain jurisdictions where security professionals are personally liable for decisions being made. Uh I think the UK is a little bit more liberal than that. The US is less liberal. Um so but we should all be aware that there are consequences for our actions and should take them seriously. This is a profession ultimately a very very young one admittedly not like the law or anything else but a profession

nevertheless and we should act accordingly. Any more for anymore? Yes. Good person right at the very front. This is how you learn things by asking questions. Thank you. Thank you. Very interesting talk. Um I just wanted to ask how do you deal with those sort of unanswerable questions that you get and have you found that if you get asked these questions and you perhaps try and turn it into a different question or try and answer a different question, do you get or does the client then get frustrated with you and how do you deal with that? Yeah, it's there's no there's not an easy answer to that. You're absolutely right and I was going to try and

illustrate that by answering a different question, but you you'd already predicted that. So, um, yeah, I mean, politicians do it beautifully, don't they? They kind of learn how to, uh, oh, uh, and all of a sudden, everyone's talking about something else. It's not it's not a case of deceit. It's it's a case of not taking it on. Um, and, uh, and but communicating with the client, just say, you know, that's not something I can answer for you and give give reasons why. Um, it's okay to not answer a question in that sort of polite, you know, graceful way to withdraw from the discussion rather than say can't answer that. you know, obviously that's just poor communication. Um, but yeah, trying

not to just don't don't take it on anymore. Right, you can go and get a cup of tea and a coffee. One more round of applause, please, and I'll see you in half an hour or so.