BSides Buffalo 2026: Breaking Down Your Incident Response Plan Before It Breaks Down on You

Awesome.

Welcome everybody to the afternoon session here at Bides Buffalo. This is the track one room. Uh we have four sessions this afternoon. Couple housekeeping things just to keep a note of. Please make sure your phones are on silent or vibrate. If you're going to the bathroom, please try to go outside entrances to avoid any distractions to the presentation. Other than that, enjoy yourself and uh be happy to learn. Our first uh presenter is Patrick Roast uh breaking down your incident response plan. So everyone give him a round of applause. AND all right, welcome everybody. Thank you for joining making your way here after lunch. I know uh coming back to a presentation after lunch is challenging.

So I hope you keep your attention um be kind of engaging here. So, my name is Patrick Ross and I just want to start by addressing the scarf that I'm wearing, the shirt that I'm wearing, uh, the notes in the bottom there. I presented here at Bides 2 years ago. Uh, this is the presenter scarf from that. And then I volunteered last year. That's what this shirt is. So, I just wanted to lead off with that just to encourage everybody here, whether it's your first time or you're frequent um, of Bides, it's volunteer run. I'm very proud, very honored to have the privilege to be able to stand here in front of you um to have done it in the

past. I just want to encourage all of you put in a CFB, have the opportunity to present, volunteer if it's your first year and you think it's awesome and you want to make it better next year. Uh just put in volunteer. Hopefully everybody's been friendly to you this year and you want to be that someone next year. Um so with that said, I have about uh a little over 15 years of industry experience. uh first eight plus of it has been technical roles, help desk, IT manager positions. Uh since then over six of it in security advising and in 2023 I founded info security blueprint where I help small medium businesses address cyber security compliance risk management and just be

that resource to them where they don't have one internally. And as you see on there, there's a QR code, my email address. That QR code goes to my contact card. I'll have it up again at the end. if you didn't get it. Um, feel free to connect on LinkedIn, shoot me an email, and then I have links throughout the presentation um that some you'll see on here. They're uh if you email me and you want the slides, I'll send them to you. Everything that's underlined is a link to a resource or uh something to that effect. So, when you get the slides, click on it. So, I'm going to start with the poll. And I'm curious. I'm going explain the

card in just a second. um in your organization or maybe the most recent organization you're with or if you're a student uh what you believe for your your institute uh what type of instant response plan does your business have and then the way you vote is just hold your card up and I need to switch to it. Give me one second. Um hold your card up, put the letter towards the top and then that'll capture your vote when I walk around in a second with my phone.

this right up the graph as we go.

>> Oh. Oh, I see. All right. >> Results are coming in technical or combined. Okay.

>> If you have two different ones, it might not be the same. But >> they did only 30 cards. So, sorry. I can do this now. Okay. >> So, overwhelmingly combined uh some technical someone business only and a couple neither. But

>> and I take this off while I finish presenting. It's getting warm. Um, I want to start with that just because anyone who's in the technical incident response, who's who's actually um in charge of the technical instant response by a show of hands for a business? Okay. >> Is it is it half? Okay. All right. So, one and a half people. >> Um, you're trying to find out how did you get in, what did you take, right? That's summing it down, making it very simple. Did you realize how much you have in common with the song Cot Joe? No. >> Where did you come from? Where did you go? >> I need to troubleshoot that cuz it was

supposed to be sound and the next clip really relies on the sound. So, give me one second to figure out. >> All right. Okay. I'm glad that works for the next one. Um, today I'm going to talk more from the business side from what are the business decisions, the business actions that we have to go through and not as much of the step-by-step technical stuff. Um, the technical is going to be happening in the background, but as you're doing your job and as we're working in the business, how do how do we have to respond to that in the business side? And just a brief touch on instant response versus business continuity versus recovery. They are three

different plans. If your business organization is small enough, maybe you combine that them into sections of a plan or you merge this continuity disaster recovery a little bit, but they really do have different functions. So instant response is how the business is going to react to potential events and incidents. Uh we'll talk about events and incidents in a second, but notice potential. It doesn't have to be, oh, we have a breach, it's an incident. It could be we think something's happening, let's investigate it. And then the business continuity is how do you deliver your product and service in a degraded state. So an incident happened, a disaster happened, we are we can't offer a service. What is our workound?

The disaster recovery. How do we recover from that degraded state? Get back to normal operating. They overlap roughly like this. That instant response starts right away. That takes up all your time. And then as that comes down, that business continuity comes up because you know the system's impacted. you know where your degraded state is and then that disaster recovery follows to bring that degraded state back up to normal operating. And then as instant response comes down, disaster recovery goes up and business continuity continues until you determine that you're going to switch back to your systems and back to your operating state. We're going to start talking about why incident response planning matters. Who recognizes this scene for just from

the still image? Anybody? >> Not a half this time, but full recognize. >> Oh, okay. All right. One person. So, this image represents your incident. Building's on fire. It's hard to see here, but you'll see in a second. Your instant responders are down at the bottom. They've deployed a tool. >> Boy, I'm in for a surprise. >> So, the speakers over here, you may not have heard. Oh boy, are they in for a surprise. So there there's your attacker up at the top of your incident waiting to come at you. So let's just watch how this unfolds. Yep, there they are. There's some tools deployed. They're ready for it. Okay, they evaded. Management's aware. Your employees are

aware. Something's going on. Someone's There's an elephant in the room. >> Attackers are happy about it.

So, everyone's scrambling for a tool, right? Here comes an attacker. Let's take our tool. And that tool has been overburdened in this failed. Was alone instant responder fell on his face. The attacker keeps going. >> You really thought about this? >> What are you doing? Another tool. Another tool. I did. I'm going to watch this again. The big head gets exposed. management falls right on their face and we're definitely making a decision. We're on fire now. Sucking up all the data. Sharing with friends of the dark web. How happy they are about it. Little little off guard, right? All right. Probably didn't hear at the end cuz we were so busy laughing, but he said, "You're making history."

>> Do you want to be the company when they make history and they attack us on the front page of the news? >> No. So, >> I apologize. Dumbled one of my favorite cartoons as a child. I hope I didn't ruin it for you. Villainize that poor elephant. Okay, so for real though, who's familiar with the Marks and Spencer and co-op uh incidents from last year? Anybody? I saw half of the hand over here. Okay, so half of half of the person in the room, a lot of half people here. Um so these are real incidents. So, you know, I made fun of Dumbo in the car, too, but this actually happened. We're going to

compare the outcome from these two. Uh, Marks and Spencer and Co-op. They're both re um retail stores and brand. And they both had incidents last year about a week apart in April. One of them was over a weekend because that's when attackers hit, right, when we're on on holiday. So, um, they were hit by the same threat actor, Scattered Spider. Probably familiar with them. and they deployed the Dragon Force malware. After the incident and testimony, Argie Norman, the Mark Spencer chairman, said that he felt like a rabbit caught in the headlights. So, when I was demonstrating this uh presentation for my wife, she goes, "Why didn't he say deer caught in the headlights?" I'm like, "That's a good

question. It's probably cuz they're in Britain. Do they have deer?" I don't know. But >> so feel like a rabbit caught in the headlights or any animal caught in the headlights isn't the feeling you want after an incident. >> That was confirmed by inside sources who said they didn't have any business continuity plan. They were figuring things out as they went. It led to employees using their personal devices, using their personal WhatsApp accounts to communicate. And there was all this outbound traffic and they were scrambling the entire time. GoP on the other hand regularly rehearsed incidents at both the board and technical levels. They just do one or the other. Did they did both they had

an unplugging plan which was critical to containing their threat. If they didn't have the unplugging plan and the segmented networks, it would have spread from where the initial attack was to the rest of their systems. Their digital information officer said that practicing how to make decisions under pressure was more useful than the technical side. So, as a result, Marks and Spencer, they didn't detect their incident for over 48 hours, and they did not have segmented networks. Those are both technical failures. They had plans that were not tested and no instant response retainer in place. Both planning failures. As a result, they couldn't deny paying the ransom. They had six weeks of outage in their online store, and they were pen and

paper at all of their brick-andmortar stores. over $400 million equivalents in loss, over 130 million in response to recovery costs alone. They end up tripling their cyber headcount, doubling the cyber spend, and their chief technical officer left. They uh lost some of and they didn't disclose how much, but some of their 32 mill uh million customers data. It's unknown how much, but some. Co-op on the other hand, they detected in less than 24 hours, had heavily segmented networks, were able to use the unplugging plan. both technical successes and then they had the detailed plans, the regular test and a forensic contractor relationship they're able to lean on immediately planning successes. Because of this, they didn't pay the

ransom. They had brief almost unnoticeable online and intore disruptions. Still had 107 million uh dollars equivalent loss and 27 in response to recovery cost. That's about the quarter of cost of the same attack that hit Marks and Spencer. and they ended up losing all of their six and a half million um data members or you know members data was stolen. So still had an incident, still lost data, still paid a lot of money cuz that's going to happen if you have an incident um potentially but it's not as bad as if you don't plan and you see that impact that hit our dispensary. Moving on to the core components of an effective plan. First step is to define what an event in

an incident is. How do we know we have to use this plan? So if you see something like laptop is running slow, is that an incident? Until you look into it, you don't really know. Probably not. They probably they might just have software running. They might Windows updates running. How often does that slow down computers? You hear about it, right? If it's that it's running slow cuz malware not as being encrypted. If you look into it, it could become an incident. An employee needs a password reset. They probably typed it in wrong, right? If it's five employees need a password reset because they all had someone trying to log in at the same time, it's an event. Maybe it becomes an

incident. So you have to define where it is each of those. And then you have to have your parameters and then your responses for each of them in your plan. So if we have an event and you want and you different severities of events, if it's a low-level event, maybe it could be handled at the IT manager level. Once it escalates to another parameter where it might be a medium or a high severity uh incident, that might be where you have to get the CFO involved or the CEO involved. People who might have to make higher decisions, make different um communications internally, externally, have resources that you might not have at the lower levels. And that should be

defined ahead of time because you don't want to be scrambling and figuring out who's responsible for this. There be a lot of people competing to try to take care of things, pointing fingers, not wanting to take care of things. If it's in your plan ahead of time, you'll know. So on the response side, how are you alerted that there's an event or an incident? Probably ticketing system, right? It could be through a SIM. You might have alerts coming from an individual device without a SIM. Can your employees send an email to incidents ad and report something that you would look into? Once something's reported, it could be an event. After you look into it, you might

determine, oh, this is more than we thought. It's an incident now. And then you could set a priority on it. And that could change. It could be an event when you start and it could be a higher level priority when you end. That event might be resolved or it might be a non-event. Might just be another break fix ticket. So having a response plan like this and knowing where we are in this for everything that comes through and how do we categorize it? Do we have category uh categories in our ticketing system? Do we what's our process look like is important. That way um incidents don't slip through because you miscatategorized it. Document in your plan who's responsible

for which actions. Some examples. Who can classify as an incident? Sorry about that. Who can classify something as an incident? Is it your initial um help desk technician who's seeing the ticket that can recognize it and go this is an event, this is an incident, this high level incident, or do they have to pause and say, I need to go talk to my help desk manager. I need to go talk to to um the IT manager who's able to classify it. As you get to a higher level, you might you might just we talked earlier about having uh defining the incidents, you might need someone else higher to classify it there because of um they need to be aware and

then who manages the response again? Is it that same same person or do they need to reach up at which level are we managing that response? Once it gets to a high higher level of priority, someone higher in the seauite might want to manage it. communications internally and externally. We'll talk about in a second, but same thing. And important for me is to include that contact information right in the plan. You shouldn't have to go to Outlook and to a directory and search for a name and try to figure out uh where's our organizational chart, who are we emailing. Anybody who's involved in instant response and has a role of responsibility should have their name,

their email, their phone number, and we're going to talk later about if there's an out of band communication method method, and that should be in the plan. So, you look at the plan and know everybody that's involved and how to get a hold of them. So, out of ban methods allowed, we heard in that Marks and Spencer scenario that employees were using their private WhatsApp. If you plan ahead of time, you can have your own chat method listed in the plan. If email's compromised and we don't want to use it or it's unavailable because that's part of our response, we're not going to use Slack. We have company Slack, right? If you have a method, um, are you going to say in your

plan, if all of our communication methods are down, please text each other. Like, let's figure it out. Like, you can put the authority to use personal communications in, but you can also have a way to put in backup systems of communication. And then the other part of communications, controlling the narrative before the rumor spread. Internal, you want to reassure your employees where you're at. we're investigating it. It is impacting this system. It is not impacting this system. Um they're going to be the ones that are going to turn around, have the external communication. So your employees need to know when we're talking to clients. We shouldn't say, "Oh, we're having an incident. We might be breached, right?"

You know, oh, hold on. Don't say that. So control what your employees are saying to clients and partners uh people they're communicating with and make sure that you have that unified meth message going out and document in your plan who can authorize different communications and what methods you will put communications out uh including regulatory requirements at what point do we need to tell somebody that's one of our regulators that we have an incident how many records have to be lost >> so one of the challenges with using an outofband communication mechanism is um the lack of privileged communication, right? So, this becomes a legal event very quickly and you're going to want to make sure that anything

that you're sending between members of the team >> falls under privilege and if you don't have a way of actually categorizing that >> then it becomes discoverable, >> right, should you have to go to court. >> So, >> yeah, that's a really good point. That's why having a secondary method established and instead of an out of band is important because if you don't have that, it's a free-for-all. They're going to do what they want to do. So having something established is a better method right? >> Um yeah, no, I agree with that. >> Uh I think so on the regulatory requirement side, how many records before we have to report to HIPPA, to FTC safeguards, to whoever your

regulator is if there's an incident shield act. um cyber security insurance, you should know when you should be contacting them in this process. They usually want to know first and who your contact is. And again, have that info in there so you don't have to go look up your if you have access to it, your cyber policy. You have here's who we contact, here's the company, phone number, email, uh and maybe a little bit of information about your um policy in there. That way you can reach out to them and you can start the conversation right from your plan. forensic response. Hopefully, this isn't the first time you're calling someone saying, "Hey, we have an incident. Can

you come help us?" I think they'll probably assemble pretty quickly and come come see you. But if you have a relationship with them beforehand, they have an idea what your system is. They know who you are when you're calling. If you're not showing me with them, at least familiar with your name, uh it'll go over a whole lot better. And one of their concerns is going to be, how are we getting paid? Can you pay us a retainer? If you've talked about that ahead of time, you'll know how to do that and you'll know they can trust you and not just somebody calling out of the blue and uh you don't have any help pulled up there.

Law enforcement. Are you going to contact law enforcement? When do you contact law enforcement? And then media, it could be channel 7 calls. This could be what are we putting on social media on our website? If somebody calls, have an idea of who's coordinating that response, what kind of messaging you're saying, document it. So, I touched a little bit. So, document the mandatory report requirements, the time frame, and the contact info. I mentioned that before. Parameters for reporting a crime. I touched on a little bit. Um, and the pre predetermine if you'll pay a ransom or the decision tree of pay ransom. Um, if you have a hard note, it should be in your policy. If it's a

we're going to do everything we can to avoid it, but if we need to do it to recover because our backups failed, you know, there's an exception, that should be documented ahead of time. Um, having a caveat on there, you'll pay a sanctioned country or not or actors who are sanctioned, you you want to have that conversation ahead of time and and have that documented somewhere so you know if you're hit with ransomware what your response will be. You're not it's not the first time you're discovering it. >> Yeah. And I'm guessing that it's kind of implied in here is that legal counsel should be part of these incident response plans. Yes. In review and the

tabletop exercising of it. >> Absolutely. Yeah. I I should probably have them in that line of external contacts or internal or external depending what it is for your business that they should be documented and have contact and they should be guiding all of this >> legal. We usually have legal and communications teams have pre like there's boilerplate messaging for different tiers and stuff. >> Yeah. Okay. >> Yeah. Yeah, for communications it might be you might have a marketing manager but maybe it's not communications but they they're in charge of it. If it's a large enough business you might have somebody who's communications that works with uh messaging and there are out outside PR and communication firms. So

if you have an HR manager who kind of does marketing but isn't comfortable with the outside PR that could be an external contact as well. You know the point about legal absolutely because that's a question we want to ask this about ransom about paying a ransom we go out of business or we pay a ransom to a sanctioned company or country or sanctioned group that there are criminal and civil penalties if you do that. So you have to make that decision if it's if it's a risk you want to do or not. >> Patrick. >> Yeah. >> The other thing we have legal do is they manage the cyber insurance relationship for us. >> Right. They're the ones who are going to

make sure that you're doing the things that the insurance policy is going to going to cover. We've been in situations where that doesn't happen and then cyber insurance says that was an unapproved expenditure and we're not going to cover it. >> Yeah, that's important. There's a lot of um phrasing in cyber insurance. You want to make sure you're doing the right thing and legal can make sure that you are at least to their happiness and their assurance. Yeah. documentation. Uh, where are you keeping the details of your incident? If you have that ticketing system or a SIM or something that's that's kind of controlling your process, it's probably in there. Um, if you have documents that you can't attach

or you can't put into your ticketing system, is there another location for it? And then, what type of information are you collecting? Do you have a templated um evidence collection form either as an appendix of your plan or as another document that sits with your plan that says here's the details we need make sure we do it and fill out that form to make sure you didn't miss something cuz you're scrambling and under pressure and how is evidence handled. Um this is something you want to you know talk to legal with a little bit if you're engaged with law enforcement hopefully you are ahead of time or your forensics response they could tell you this is what you need to

do before we get here. Please don't do this and let us do this part. Um they'll give you that guidance on it. So >> yeah, >> and this is something I I wish regulatories I mean I kind of don't wish that they would do it, but I don't understand why they don't. There's no criteria in evidence in any type of evidence collection to do forensics. So I just need to prove that I have logs. I never need to prove that I can recreate from those logs. All my regulatory bodies like you know DYFS or Sockley, they're going to want that. But no one ever asks uh you know they just ask about integrity but they never ask how

about recovery and I think that that's something that a lot of organizations miss is they should regularly test their forensics capabilities uh postb breach. So >> yeah I think that's a big step that that could be missing. Yeah. And it would cut down on the cost when you do that kind of. Yeah. >> Yeah. Coming from the response mindset, um you're focused on the containing the incident, recovering from it, and removing the threat that a lot of times it's uh you know, delete it, flash, and clean it and get rid of it. But that might not You have to do it to a certain extent, but that might not be what you want from that evidence collection

standpoint. So, >> don't delete the keys, just >> Yeah. Right. Um I see this. So I'm also a volunteer firefighter. So I see that here is our initial thing is preserve life and preserve property. But then once we know we're in a comfortable spot, the fire ex fire investigator is going to come there and say, "All right, well stop tearing stuff down. Stop stop stop opening walls. Like we need to go look and try to see if we can figure out how it started." So you know, we need to address that incident, but then we need to slow down and keep the incident under control while they're investigating and then we can continue and make sure

everything's extinguished. So, uh, same thing here is you want to make sure that you're going as far as you need to to protect your system and then know know where that line is. Some common pitfalls and how to avoid them. Your plans too vague or too complex. So, if it says upon confirmation of IT incident, it will notify appropriate parties as soon as reasonably practical. See appendex B. Who are appropriate parties? Why can't we just put a name in there, a title in there? Um, I like I like titles and policies and plans more than names. I think there should be those roles and responsibilities. You name them and the rest of the document should be titles, but put a title in

there. We'll notify security director whatever your title is. And then as as soon as I don't think that's good as I I think as soon as 1 hour, you might think it's 24 hours. So, put in there a time frame. Um, and this goes back to categorizing your incidents. If it's a lowle incident, maybe it's 24 hours. If it's a high level incident, it might be an hour. It's kind of the same way you would treat a vulnerability um matrix of how vulnerable how what's the ranking of that vulnerability that should indicate your response there. Too complex. Probably can't read a thing on that, right? Um, if you have a flowchart that looks like that and we need to sit

there for an hour and decipher what the next step is, you probably made it more complicated than you need to be. Um, and this is probably an extreme example of it, but if it's seven pages of how to classify an incident, it's probably more complicated than it needs to be. Um, especially when things are busy and you don't feel like you have the 10 minutes to sit down and read those pages. You're going to skim through and you're going to miss stuff or say, "Never mind. I'm not using this. I know what to do." So don't make it overly complex. Find that sweet spot where it's prescriptive but it's not cumbersome. If you don't have organizational

alignment, you're going to be working against the organization in your response. Who's familiar with Colonial Pipeline from a few years ago? Okay. Like three times as many hands as my other questions. I like it. So the thing that happened with Colonial Pipeline was their systems were compromised. They restore them from backup. They're successfully restoring from a good backup that they do works. CEO said they paid the ransom because the company wanted to accelerate the recovery time. Do you think that the people who said that the backup ever talked to the sea suite about what the recover accepted recovery time is? Probably not. If your recovery time is 3 days and your CEO says we need to be back up in a day,

we're going to pay multiple millions of dollars ransom so we can come up quicker. you probably don't have the right systems in place and you need to go back to leadership and say if you can't be down more than 24 hours, we need the investment to to change our tools to make it better to achieve what you want. And they probably could have done it a whole lot cheaper than I think it was $8 or $10 million they paid for that ransom. You could get a lot of backup solution support for that amount of money to be proactive to not have to pay a ransom. Um, so make sure you have organizational alignment on your

recovery points and your recovery time objectives so that you're going you're not losing more data than you need to and you're back up in time. Humans, and no, I'm not blaming humans. I'm blaming people who don't train the humans. Do the employees know they're listed in the plan? I said to put titles and contact info and maybe personal um cell phone numbers and stuff in the plan. Does your CF know CFO know that their name is listed as the person who's going to be the incident manager for high priority incidents and their cell phones in the plan? Because if they get a call on the weekend and say, "Hey, we have an incident." And they don't know it, why

are they listed? And then if they're listed, if they train to execute the plan, they have to know what the plan is. They have to know the role in the plan. And they have to know, oh, you're calling me because I need to now contact contact legal counsel. Okay, that's my my um part of this plan. That's what I need to do. And that'll lead right into testing and maintaining the plan.

When you're designing tests or tabletop exercises, make sure they're realistic. So that means don't go overboard on World War II broke out and there's a fire and instance. Okay, hold on. Slow down. Right? Make sure it has to do with your business, that it is um industry aligned. So, if you're a manufacturer, look at the CIA priority. You probably care about availability more than anything. If you're down for over 24 hours, you've lost a day's worth of production and a day's worth of revenue. Your instant response plan um test should be designed towards something that takes away your availability. If you if you work in a highly regulated um information field like HIPPbooks, you have health

information, you probably care about availability as well because every business doesn't extend that confidentiality might matter more because you don't want that breach of health information and to pay for all those records lost and to have lawsuits against you for losing them and all that. So make your test around a breach of sensitive information. Make sure that you're not doing a test of your onrem Exchange server if you migrated to hosted email last year because it would be a useless test. Um, if you made a change, it's a good indication to do a test again. But make sure that you're testing what your cloud systems are. Anything that's hosted, you can test the compromise or loss of

availability to those systems. And if you're hybrid, you can uh you can test that as well. Just make sure it's it's focused what your business has and they should be risk based. You probably only have one chance to do a test in every period of time. It might be once a year that you can actually get your leadership or your executive suite together to do a test. So if you have to pick the one, what is the most likely the biggest impact risk if you pick between those? Don't just do some scenario like, "Oh, this would be a fun one to do." do the one that if this happens tomorrow, it's a problem and we

need to make sure we're ready for it. And if you're lucky enough to do a second one, then you can pick the runner up and do the second one. >> Your pleasure. >> Yes. >> One of the mistakes I made early on was I would do a tabletop exercise and I would have the IT people and I would have the business line people in the room together at the same time. >> The problem with that is is that I'm testing different sets of skills, right, based upon your function in the IRP. If you're part of the IT team, you're very technical and I'm going to when we present the scenario, I'm testing your ability to identify, right, detect,

right, analyze all of those things. >> If you make a CFO sit through that part of it, you got a different problem because they are completely checked out at that point. >> So, what I tend to do is I tend to do two the same scenario just run it as two different groups because the skill set that you're going to be testing is very different. When you get to the executive side, you're talking about decision-m how do do we know how to make this decision, right? Who has the authority to make this decision? >> And it isn't, you know, what functions do you need to perform, right? To to return service. So, um, once I made that

change, it's easier to get buy in and you can contain the conversation a little bit better to the scope of the attendees. >> Well, I agree with that. So I was actually talking to an IT manager recently and I said I think we'll do a technical one first and then your responses from the technical one you I'll have you also in on the other one >> but we're not going through the technical decision making and when they ask a question of like oh how long have we been down or what's the response we'll have predetermined the answer you can provide to them. So I do I definitely recommend doing them separate. um or if you don't do them

separate, at least make sure that you have a very short and narrow scope of how far down that technical rabbit hole you're going um in that meeting. So, I thank you for that.

And when you're doing these exercises, make sure you have lessons learned. Whoever's facilitating this, whether it's if you have an external company come in, whether it's an internal resource that knows how to do it, they they should be taking probably most of the notes because there's different things that they've designed their test to take notes on, but any everybody should be doing it, submitting it back to that person. If the CFO says, "Wow, I felt uncomfortable here." Or, "That's not how that process would work." They should take those notes and they should notify whoever's organizing um organizing the task. So, and that way you can go back and update your plan after. And not only is it from your

simulated exercises, do it for your real ones. If you if you have to use that instant response plan, have a lessons learned after and don't close out that incident until you're done. Obviously, work on it and resolve it, but leave that ticket open or at least leave a follow-up task open and don't consider it resolved until you've gone through and said, "Okay, that was crazy. What went wrong?" you're never going to test it better than when you have an actual incident, even even the simulated test. So, take advantage of having the lessons learned from a real incident. And you probably don't have those highle incidents as often as you might you might actually kind of regularly have

low-level incidents that you're resolving pretty quickly and use those. Those are great tests. >> Yes. >> What do you think the value is of setting up something like the chaos monkey in a dev environment? >> Setting up what? Hm. >> Something like uh Netflix's Chaos Monkey, which just goes and shuts machines down. >> Um >> so as part of the response, it would automatically shut things down. >> No, more as part more to actually simulate how people would react to closer to a real >> fault injection service or fault injection. >> Yes. Is the is the generic term? Yes. >> Oh, so that would be outside of a planned exercise. you would have that would do that and see how

>> or I'm saying you do you do a plain exercise but you do a planned exercise but rather than just doing it on the table top >> right >> it's okay we've set up the fault injection simulator in dev >> have fun >> yeah I think >> I I don't I don't think you'd ever get someone to sign off on like a ransomware simulator in dev but something like that that's closer to a real incident >> yeah I think that would be useful on the the technical side because you really that's the next closer thing below the incident besides just sitting around the table you're going to get to actually responding. So something like that

you'll get that technical experience of this went down here's what it actually looks like and be able to see it. So uh I think there's there's value in that if you have the systems to do it and there's tolerance to to to do that and the capability and everything that >> I think to to her point if yeah >> the best way I've seen fault injection systems be introduced is not to get buy in to test something that already exists but to culturally build a program where if you're developing a public facing application or anything holding critical customer data you have to build a representative version of that that we can do fault injection into

>> and that just becomes part of the security review. I get to go in and try and break it and yeah, >> I've also seen that with database backups where if you can if which is more relevant to the ransomware scenario where it's basically any database that you set up, you have to be able to you have to have a backup and restore process which can be tested. And to to do that, even if it's just a test in dev, you have to either be able to test the backup and restore process to a different environment or you have to have something that's representative that you can test. >> Thank you. >> Yeah. And then reviewing the plan,

the plan itself should be reviewed at least annually. Most regulations require this. Most external, you know, cyber insurance, mo anything that you have coming in requirement wise will probably uh require this. So, um, do it annually. If you have significant changes, like I said, that that on prem to cloud change, test it. If you have, um, employees that are named in there that turn over, update the plan, test it. That should it should really trigger, hey, we need to make sure we update it and then the people involved know and what they're doing with the system. So, anytime anything significant changes, that plan should be updated. Um, update it anytime you have to do those lessons learned. We

just talked about that. Um, confirming the name, the contact info is current. Uh, so I mentioned that if you have an employee, a key employee turnover, go through, confirm that it's updated. You don't necessarily have to do a huge review of it. It could just be someone going through confirming it. And then at the bottom, I always put at the bottom, include a review log. Every time you do a review for it, take credit for it. put in there the date, who did it, if there were changes made, um, if there weren't changes made, put it in there and just say no changes or updated contact info or whatever, minor grammar changes and put it because if you have an outside

auditor come in, whether it's for something proactive like a a sock 2 or something that you're trying to get certified in or because you had an incident in your HIPPA, you're being investigated. Now, if you could bring up your plan and say, "Here's five years of histories of us reviewing it, changes made throughout." um it'll hold a lot more than you're saying, "Oh, well, let me look up a meeting from last year. We updated it." And then you're digging to try to find the proof. Put it right in the plan. You have the evidence of it already. And uh include the approver of it. Make sure you know who approved it. That way you can go to them and if there's any

questions, um the one who drafted and approved it can talk on that. It's all about um setting up a culture of readiness. So, do you So, don't let Dumbo make you look like a clown when you go back to your own environment. You don't want to be the the ones running around in the ground, right? Trying to scramble for a tool, finding a new tool or deploying a tool, falling flat on your face. And then ask yourself, would you rather if you had an incident affect your business, would you rather be Mark and Spencer or would you rather be co-op and the results that they came up with? Either way, they had an incident, they

lost data, they lost money, but you can control the size of that impact by preparing at the time. With that, I'm going to end on here. My contact info's back up. if you want to scan the card upper right hand coer. Um, but please, please, please before you leave, scan the feedback uh code and be honest. If there's something in here that you like that you would want me to conclude the presentation again, I want to know about it. If there's something I was wrong about, I didn't cover that you were hoping to hear today. Um, put it in the feedback. Um, I do have this presentation uh scheduled two more times. I'll be besides Pittsburgh next

month and I'll be infoscent Nashville in a couple months. So all that feedback is only going to help me for few uh future versions of this presentation. So thank you very much. Appreciate your time. Appreciate your feedback.