The Challenges Of Remediation During A Cyber Incident by Johanna Strachan

Thank you. Um so thanks everyone for coming to my talk on the challenges of remediation. A bit about me is that I work at PwC in the instant response team and I primarily focus on instant readiness. That's kind of all about creating technical runbooks, instant response plans, and assessing SOCs. In my spare time, I'm also a postgraduate student at Royal Holloway studying for a master's in information security. And this talk is actually a really condensed version of a much larger piece of work I'm doing for my master's. Um so it is a work in progress, so please be kind. The methods I've used to carry out this research include interviewing cybersecurity professionals who have an

element of remediation in their role and reviewing case studies and also drawing from my own experience. And in this talk, I'm going to discuss the strategic, technical, and broader industry challenges with remediation. So first off, I thought it'd be really good just to discuss the definition of remediation. I have interviewed many people and asked them, "What do you think and feel remediation is?" And as you can probably see, I got a really wide range of answers. So on the left, um some people believe that remediation is just purely about eradication um and they actually use the term eradication and remediation interchangeably, which I think is incorrect. I think eradication is really just about the action of

getting the attacker off the network, whereas remediation's much more of an umbrella term. On the right hand side, you can see that some people felt that remediation is much more of a longer-term kind of transformation project that could go on for months and months. And the key takeaway here for me is that remediation can be many different things and it's really dependent on the incident that you're facing what kind of remediation you'll take. So first off, I just want to talk about the strategic challenges of remediation. And I think a lot of people probably sat here might resonate with the first two challenges I'm going to talk about, which is money and cybersecurity not being a priority.

Quite often you get organizations who either don't have much money to spend or they do have money to spend, but they're just not going to invest in cybersecurity. And that means that when they're experiencing an incident, they've probably cut corners and not invested in a great EDR tool or other logging and monitoring capabilities and they just can't see what's happening in their environment. And this means that when an incident does occur, the investigation swallows a really, really large chunk of money they never envisioned having to spend. Um and a bit of a discussion point is actually how much should remediation cost? And a general rule of thumb seems to be that you should be prepared to spend the same amount of

money on an incident on the investigation element that you're going to spend on remediation. So that can be really, really pricey. And this is a really unattractive proposition for some organizations and it often deters them from carrying out a really well-detailed and thought-through remediation plan. Moving on to lack of preparedness. In my experience as a practitioner, I can't think of many instances that I've been involved in where you rock up, you speak to the organization, and they say, "We've used our incident response plan, we've referred to the incident runbook, and everything is just going really, really well." Um and quite often that's because these materials don't exist. Organizations haven't invested in preparedness. And this can make

remediation really, really difficult as they're probably thinking about time, you know, what are our business-critical systems, how are we going to respond, and what are our priorities. And this can all lead to errors or delays within remediation. Lastly, I've put lack of understanding. And what I mean by this is that senior execs are often the ones making the big decisions in instances. Um but they actually also don't probably fully understand or appreciate the scale and complexity of an incident. And I think this is because they often don't have a cybersecurity background or training in in cybersecurity and that's where their limited knowledge and understanding comes from. And this is a challenge because they're therefore after quick wins and the

cheapest solutions and they're just not interested in a really well-thought-through and probably pricey remediation plan. Okay, so next is technical challenges. I think it's really common to see organizations not fully understand their IT estate, which is incredibly problematic during an incident. They either have, you know, like an incomplete CMDB or a bunch of systems in a different territory in a different office they never knew existed. And this makes remediation really difficult because you need a complete picture of what the state is in order to restore it to its known state. Failure to execute the technical requirements of remediation. I think this is a technical and strategic challenge. I think it's a technical one because organizations sometimes struggle to have

or find the expertise to carry out the remediation actions. Often you'll need cybersecurity professionals, a security architect, maybe even security engineers. And we're in an industry that's already facing a massive skills shortage, so finding these people is incredibly hard. It can also be a strategic challenge, too, as organizations often decide to cut corners and do the bare minimum technically required of them. So this results in them not carrying out the full remediation plan, leading to potentially a less effective remediation process. Not allocated time and resources in advance. If the remediation exercise isn't scoped or budgeted correctly, it's probably going to run into problems. Um and for example, with an APT intrusion, you're probably going to want to observe

them for quite a while and use the findings from the investigation to feed into the remediation plan. And you need to have people available and ready to carry this out, which quite often organizations don't. Lack of validation. Victim organizations don't tend to test or um validate the remediation actions in preparation of an incident. And so if you think about it, with normal dev-to-production scenarios, we would test these. So you would test blocking a domain on the proxy beforehand. However, when it comes to remediation, organizations don't often do this. And so, you know, when it comes to the eradication weekend, it's probably the first time they're really trying to attempt a mass password reset. And so

it's probably quite likely something's going to go wrong and the attacker will come back in. I also mean a lack of validation in terms of like plans and processes. Um I think it's really great if an organization has an incident response plan or runbooks, but if these have never been tested, um the value that it can bring is probably incredibly limited. And lastly, incorrect approach. Um sometimes organizations quite like the whack-a-mole approach, which is basically where they identify a compromised system and they take that system offline and they just repeat this over and over again. Um what this is doing is just luring the victim organization into a false sense of security as they're not actually

remediating, they're just simply pushing the attacker into a part of the environment that they can no longer see. So I now just wanted to tie together the strategic and technical elements I've talked about by looking at a case study. And I'm going to look at um the breaches that the Office of Personnel Management or OPM suffered. Um OPM are a US government agency that holds a lot of personal data. Um they basically have data that's used for background checks and security clearances in the US. And they experienced two separate cyber incidents in 2014 and 2015 that were thought to be linked. And I will say this is a publicly known breach and this timeline has been massively condensed

because the two breaches were incredibly complex and lengthy and I just wanted to focus on the remediation part. So you can see that in November 2013, threat actor group one in orange, they gained access to the environment. They hung around for a while and then threat actor group one exfiltrated some data, nothing too big, but obviously not great. And OPM eventually realized and they planned a big bang event in May 2014. However, because remediation's really challenging, um OPM failed to notice a second group of threat actors in pink. And due to OPM not fully understanding their IT estate and also not prioritizing cybersecurity, this second group managed to remain in the environment post-eradication event. Um

they managed to create a backdoor through some malware, which was undetected. And unfortunately, it was actually this second threat actor group whose actions resulted in 21.5 million um individuals having their data compromised. So the point I'm trying to show here is that a poor remediation process, even with an eradication event, can have a significant impact. The impact here being that the failure to detect a second threat actor group meant that millions of individuals had their data compromised. I also thought it'd be really good just to do a bit of a deeper dive into some the key failings that OPM had. So OPM leadership failed to prioritize cybersecurity and this is very much a strategic failing as OPM had been warned

about their poor security posture as far back as 2005. And unfortunately, even after the security breach they realized in 2015, OPM still failed to prioritize cybersecurity and implement recommended controls. They also failed to execute technical requirements and this is very much a technical failing as OPM did not implement two-factor authentication back in March when they learned that threat actor one was in the environment. And if they had, they could have either significantly delayed or even mitigated the second data breach discovered in 2015. Lastly, I just wanted to talk about some of the broader industry challenges that I came across whilst carrying out this research. I was really, really surprised to see that there's no remediation

framework in place. So, there is nothing stuck like outlining a high-level kind of best practices or just the key things to do or to consider when remediating. And I think this probably contributes to organizations not scoping their remediation correctly and also not being prepared. I think if organizations knew what kind of remedial actions they were likely to have to take during an incident, they could test and exercise this and be better prepared. There's also no remediation courses and this contributes to the issue there being a lack of hands-on keyboard keyboard expertise to execute remedial actions. A formalized training course would probably create a bigger pool of experts and ensure there's an industry baseline standard of knowledge.

And lastly, I just don't think we talk about remediation enough. I think we talk a lot about incidents and investigations and just general lessons learned, but we never actually speak about remediation and if it went well or if it didn't go so well. And I do fully understand that remediation is not a very glamorous or sexy subject to talk about within cybersecurity, but I think there's probably a great benefit of just sharing about what went well and what didn't work well. It's probably also worth mentioning at this point that I'm fully aware some organizations do remediation and they do it really, really well and it's successful. However, unfortunately, it's a challenge in itself to find details on

this because again, we just don't talk about remediation even if it's gone well. And that is the end of my talk. If you have any questions or you just want to discuss anything that I've spoken about, please feel free to get in touch with me and I'll also be hanging around here afterwards, so come SAY HELLO.

YEAH, SO THAT'S INTERESTING. I guess I still think there's benefit to talk about it and I think, um, you know, when I talk about things like there being a framework and, you know, what would that kind of look like? I think it should kind of be at industry level so that anyone, if you're coming from IT or cybersecurity, you can refer to it and learn from it. Um, yeah, but that's an interesting point.

Mhm.

No, actually. No, it's not come up yet, but that's really, really interesting.

I mean, obviously that makes it very, very difficult. Um.

Yeah, and maybe that actually goes back to the framework point, you know, if we had best guidance on how to remediate with legacy systems in mind, that could be really, really helpful, I think. Yeah.

Ooh, good question. I would say, um, definitely communicate, speak to everyone, speak to kind of like the NCSC, speak to everyone in your organization and seek help. Okay. Thank you very much. Thank you.