Don't Roll a One

I would like to introduce our next speaker who is Tyler Hudak and he is going to talk all about you know not rolling a one at least some of you must get the joke at least I hope all right and with that Tyler all right give me one second here so this is kind of an important question I I guess for uh this talk is how many people do understand uh what rolling a one means how many people in here play D and D all right perfect so uh at least half of you so hopefully I'm not going to lose uh most of you on this um but if you don't know uh in role

playing games and many role playing games including Dungeons and Dragons when you low when you roll low that's the you know a bad thing if you roll a one that's called a critical failure and so as we'll kind of get to this I'll kind of like Circle things around as to how this is actually related to tabletop exercises uh specifically instant response tabletop exercises um you know well let me let me just dump into it so uh my name is Tyler Hudak there are two things you should know about me uh the first is that I lead the amazing trust SEC incident Response Team a group of amazing individuals that uh we help uh clients when they get hit with uh

attacks go in help them recover help them figure out what happened and so on but incident response is an interesting industry because you don't have work all the time it comes in waves and so you have to feel that downtime and one of the ways that we feel that downtime is through scheduled engagements and our most popular scheduled engagement is uh tabletop exercises I'm also a huge gaming geek um you can kind of see it in the uh the picture there in the background that that shelf behind me goes the entire wall and is just completely full of role playing games I'm a huge RPG geek uh I've actually been publishing some books trying to figure out how my retirement

job can be to like professionally get paid to do that which I think would be awesome uh but I found that IR tabletop exercises are very similar to roleplaying games so let me go into that U my contact information is uh on the bottom it'll be at the end too all right so three things that we're going to cover first what is an IR tabletop if you're not sure what it is we're going to talk about that just so we're all on the same level the same page as to what it is uh the second is uh what are some of the mistakes that I've seen when you set up a tabletop so there are really two phases you go

through when you run a tabletop or when you do a tabletop uh the first is you go through you go through the process to set up that tabletop exercise and it's not just a hey we're going to do a tabletop exercise next week there's a lot that goes into it uh so and I should have probably started off by saying this I've run tabletop exercises for at least a decade now uh the first time I ever ran one I was freaked out because our client had asked hey can you run a tabletop and back then nobody really did that nobody really talked about that there were very few resources online and so as I was sitting there at

my desk freaking out trying to figure out how can I get out of this engagement uh and not have to do it I kind of turned around and saw my shelf of uh role playing games and it was much smaller than the the one that was in the picture before but it just kind of clicked then uh it that's when I realized that a tabletop exercise was really no different than an IR I'm sorry a tabletop exercise was really no different than a role playing game or run a role playing game and I could do that I've done that for many many years um what I'm going to try to do here uh and I know I'm kind of jumping around a

little bit but uh is really just show the parallels between you know running a role playing game because I'm how many people in here have run a role playing game before all right cool a lot of you you you could absolutely tomorrow go and and run a tabletop exercise with that experience um but there are definitely some common mistakes that I've seen in the 10 years that I've been running uh uh not only uh role playing games but um tabletop exercises uh we're going to talk about the mistakes that I've seen when you're setting one up and then also the common issues that kind of come out of tabletop exercises because after you run it you're going to have these gaps

that you're going to find you're going to have these Lessons Learned I'm going to tell you what the common ones that I see are so then that you can go back into your organization and say hey do we do this or do we do that uh and kind of you know make yourself look good and you know take care of those before you even go through an exercise we are definitely going to get a little nerdy here so I apologize now um if uh if you're not into D and and all that uh just ignore that that stuff just out of curiosity how many people have already bought tickets for The Dungeons and Dragons movie just me all

right all right okay so what is a tabletop exercise really a tabletop exercise is just a way to test your incident response plans and procedures because you all have incident response plans and procedures right sure um but test them before an incident occurs uh the way that you run this uh or there there are lots of different ways that you can run tabletop exercises the traditional way well first off why um you know first you're you're running these to see if you your organization can respond to an incident uh actually I'll pull up all these here oh that was bad animation um you want to test to make sure that your organization can respond to that incident you know do

your plans and policies work like you expect them to uh you know a lot of times we write down these plans and then nobody ever tests them till the the event actually happens and some of the assumptions that were made when writing those documents aren't true or maybe have changed if if your plan was written two years ago your organization is probably different and so you definitely need to go in and test that uh you also want to make sure that everybody knows what they're supposed to be doing um I have been on many tabletops for that I've run for clients where the organization we're talking to the organization either business units or it or security and one person says oh well

this group would handle it and that group just goes pale they have no idea what they're talking about um or especially when you talk to the business side especially senior Le leadership they probably really don't understand what they're supposed to be doing in an incident uh and and we'll talk about that uh in a little bit so the whole goal of this is to make sure that everybody knows what they're doing and finally making sure that any third- party providers that you utilize also understand what they're supposed to be doing and what you expect them to do is actually what they'll do I I'll talk about an example of that later on so how do they work uh first you've got the

moderator uh or moderators uh usually one or two uh and this is going to be the person or persons who are describing the scenario to the uh the participants um the scenario is going to be a mock incident it could be a ransomware attack it could be a fishing attack it could be an Insider threat it could be a combination of all of these uh but the moderator will describe the situation and the participants just describe back what they're doing this is a traditional tabletop exercise everybody's kind of sitting around more often than not nowadays it's that virtual table in Zoom or teams or something like that and everybody's just kind of describing what they're doing the the job of the

moderator is to describe that scenario to them and then dig into it if somebody tells me that oh well we were just restore from backup that's great I'm now going to throw in and inject that hey your backups have been deleted by the attacker first off tell me why that can't happen and then I'll make a decision as to whether or not it can actually happen and if it can you don't have backups anymore now what are you going to do and then you you kind of throw the moderator throws these things at them again and again this is a traditional one you can absolutely get more Hands-On uh with the tabletop exercises we've done some that were um

where we'd actually ask the uh participants especially on the the it security technical side to go out and hey you told us you're going to go contain this you would contain systems there's a system over here that we've set aside specifically for this go do that uh and then you know make them actually go and do that and then when that once that happens their next step would be oh we would just pull data off that great this system's contained go get me that data you told me you would get and often times when that sort of thing happens it's like oh well I can't log into it because it's contained and I we only have domain or active directory

credentials to that and it can't connect to active directory to validate that like great now now what are you to do cuz this is you know you start to get a little bit more and more Hands-On I will talk about a situation though uh later on of of how far you should go with that Hands-On uh because I have definitely seen some issues with that um but this should look a little familiar uh you know again you know you have the moderator the the dungeon master if you will and all the participants who are the players you know you're describing the scenario the what's going on they're describing what they're doing back to it um very similar

to Dungeons and Dragons uh back doors and breaches from Black Hills very similar to that in fact that's thank you uh that's exactly what uh back doors and breaches is it's a great way to you kind of do one of the a ad hoc uh tabletop exercise and and throw all those injects in there all right um so I want to get through that a little bit quicker uh just so we can get to all the really good stuff so like I said at the beginning there are really two phases that you go through if you're setting up or if you're doing a tabletop exercise the first one is in preparation this is you know going and setting up that

tabletop exercise for your organization so the first mistake that I see all the time is not inviting the right people uh you need to absolutely 100% include Technical and business most of the time people forget to include the business side that's where the Gap lies uh you absolutely need to have the business in there because the business wants to or needs to make some of these decision some of these risk decisions during an incident um yeah uh on the technical side uh you you're going to invite any we always get asked all right well who should we invite uh on the technical side invite those subject matter experts you know the way that I typically put it is anybody who is going

to do Hands-On work during an incident either your it uh your recovery people your systems administrators uh security who's going to be doing the investigation you you don't necessarily need to invite every single one of those people because if you're in an organization where you may have like 10 or 20 people in that group invite participants who can speak on behalf of the group but invite those people so that they can respond to questions that we have during the uh the exercise and we're basically going to ask them things like how do you respond you know when you get into an exercise you don't ask yes or no questions because that's what you'll get is a yes

or no answer you ask how do you respond and when they tell you how and and I'll say this too I think I'm probably getting ahead of myself but on the exercise what of the the biggest mistakes that I've seen is uh from the moderator side is um I just completely lost my train of thought I'm I I will I will Circle back to it I I sorry um but you want to ask you know how do you respond uh you want those people there to to answer those questions on the business side you definitely want to invite the people in your business who are going to be making these you know decisions during an

incident so for example in a ransomware case who decides if you're going to pay it's not it or it shouldn't be it uh your your leadership are going to respond to that uh so you need to invite legal legal has a huge role in um uh in an incident if it disagrees with that go have them go talk to Legal because legal absolutely 100% is going to want to put things under privilege they are going to want to be contacted immediately they need to be involved in this and they're often one of the most forgotten groups uh in exercises but legal Communications Communications is absolutely key in incidents especially once you get into those really uh high priority uh

world-ending incidents like like ransomware you know you need to communicate to literally everybody that you can think of and they're going to be the ones who do that uh senior leadership absolutely they need to make those highle decisions or at least understand what's going on and a lot of times the people who are giving them that information don't necessarily understand what information they want and so getting them into that exercise to say Hey you know we're going to be briefing you on this incident what would you want to know what questions do you have for us a exercise is a great way to gather that information so that when an incident happens you automatically know

what to give them and you're not sitting there for the first 10 minutes you know listening to the same questions over and over again and finally HR um HR is really important I I would say probably more for um insid threat cases because HR typically handles Insider investigations or at least have a huge part in it but in any other incident they probably need to be informed especially if there's uh a data breach of employee information or you need to communicate with internal employees they may want to have a say in what and how you word things and so on uh so the first one was yeah who to invite the second one the biggest

mistake that I see is combining both Technical and business exercises uh what I mean by this is you should separate the exercises that you run for the business and for the technical teams the biggest reason is because the technical exercise typically goes longer and you're talking about very technical things you're talking about it you're talking about how are you containing a system how are you restoring a system legal and HR and Communications don't care about that they will get bored they will tune out and they will never want to come to one again uh so you definitely want to make sure that you run these separately yes that does mean you have to do it

twice at a minimum twice uh but trust me it is very much worth it we uh insist on as much as we can whenever we're planning out that exercise for clients that first off we're doing both of the exercises and second that they're separated out now if you have a really really small organization it probably doesn't make sense to separate them out but if you have a really small organization it's probably going to be a shorter exercise anyways but the technical I would uh you know align this more to what a role playing game is like uh you are describing the exercise they're responding to actions uh or responding with their actions just like you know

the slide that I showed before with the moderator and the participants this is very much like a role playing game or again the traditional version of tabletops you know that you can run them differently the business however you would run it I recommend running it a little bit differently how many people in here know what a session zero is all right so so some of you uh so if you don't know a session zero is a term and role play playing games where you you start that initial session and you create your characters you explain the the system you set expectations it's kind of that like intro e education about the system and the game and and

what to expect and so on when you run a business exercise that's kind of what it is because if you think about it the business side and when I say business I mean Executives leadership uh legal Communications and so on they assuming you haven't had a huge breach or ransomware attack they've only heard about it in the news and let's face it the media tends to blow these things a little bit out of proportion and so they know all the scary stories but that's it so they don't really know what to expect they don't know the ins and outs they don't know all the little gotas that they're going to have to uh decide upon uh and so

that's what I recommend making the business session like you you kind of make it half educational and half exercise and and the way that I typically approach this especially on like sales calls and stuff like that is um let's take Ransom The Ransom payment because if you ever do a ransomware exercise you absolutely need to talk about the payment on the executive side the way that we run it is first off we talk about what is a ransomware payment what does that mean how would you and we go through how might you pay it you know what what's the process of doing that why you know and the reasons why you might go and pay

it and here are also the reasons why you might not pay it so we go through that little education piece as to you know to bring them up to speed with Ransom payments and then we jump into the scenario and we say all right you've had a ransomware attack it is telling you that you can't Restore for over a week the the attackers want $10 million in Bitcoin uh they're going to release data tomorrow if you don't pay let's talk about your process for paying the ransom or your more specifically your decision-making process on if you would pay a ransom or not and start going through it that way all right any questions so far am I

am I doing good cool all right um so one question to you all what is the most difficult thing to do in D and D or just role playings in G role playing games in general everybody to the table yes that is absolutely it scheduling the hardest thing in uh role playing games is actually to get people to the table to actually play the same thing with tabletop exercises um the biggest issue that I see is scheduling to too little time uh I've had organizations want to do an exercise in an hour that is not possible I've had ex I've had exercises where I've had to talk to the executives I got 10 minutes to run an exercise not

possible absolutely not possible scenario exactly uh and yeah those are I know walking in those are disasters we push as hard as we can to not do that in the end the client is paying for what the client's gonna what the client wants I kind of have to do that but you know I'm still going to push back as much as I can so the length uh of how much time you schedule is really going to be dependent on how deep you want to go um especially on the technical side um I I'll talk about recommended Ty frames here in a little bit but the longer you schedule it the more in-depth you're going to be able to go and my rule of

thumb is I always want more time than I need so if somebody comes to me and says well we we're not sure how much time I I'll give them a time frame and then I'll say and if we don't go through all that time then everybody leaves early and everybody's happy they have some freee time to themselves you know it's always better to have more time than less so on the technical side I recommend minimum two hours all the way up to four hours I've run them for eight hours um again the technical TS tend to go a little bit longer you're going into more details and so on the executive side minimum 1 hour know I know I said

that 1 hour is not enough time I meant for the entire exercise both uh business and Technical um the business side let's face it the people talking to in in the uh in the executive tabletop it's very difficult to get them to uh work through a or I'm sorry it's very difficult to get on their schedule so if you can get an hour of their time that's awesome um so what I recommend is if you only have an hour of their time prioritize what you're going to talk about yes so if you're not Outsourcing and if you're 11 oh yeah no yeah that that would be good too because um the question the question was

uh you know can you split it out uh basically and yeah you absolutely can and sometimes that's even better to do because then you can have takeaways you give them homework and so you talk through a couple things in the first hour and then you say all right well this is what we're going to talk about in the next segment you know we had some questions from this let's try and find those answers and then bring it back yeah um the business again one hour try to prioritize uh and and when you can and that's what we tend to do we we typically only have an hour with Executives which is fine we prioritize what we're going to talk about if we

have time left over great I will tell you Executives like to go down lots of different rabbit holes and sometimes you just kind of have to let them do that within reason you know your job as the moderator is to keep it on track um I have rarely but I have had people get into fights during tabletop exercises so you kind of have to break those up every once in a while um we we do have a slide I don't have this in here I probably should have put it we do have a slide at the very beginning that says something like the moderator uh has basically ultimate power they they they can uh you

know stop all disagreements questions and disputes and so on just basically so that we have that in the beginning that says hey if you're going down the wrong path we're going to stop you um just so everybody knows all right um the scenario uh when you're writing that out and I call it a scenario because that's really what it is don't make it too difficult um right sizee it for the group um you don't need to go to the worst case scenario now um You can you absolutely can if you want to do a full-blown ransomware attack that takes down your entire network cthulu is rising from the ocean taking down everything please go ahead and do

that however if this is your first time running a tabletop or that your organization's first time doing a tabletop maybe start a little bit slower I mean think about it in role playing game terms level one you're not going up against tmat the five-headed dragon you're probably just going up against a b bunch of weak goblins um start off a little bit slowly um that being said you know when my team gets brought in we absolutely do worst case scenario because that's what we're being paid to do um but just kind of understand who your audience is is is what it comes down to um like I said start off small and build up um the big thing too is

make sure you're doing a scenario which everybody can participate in if you are having business in the uh exercise make sure that there's something for them to do don't just do a scenario which is all it and that the business would have nothing to do with that's going to be boring for them they're and and it's going to be detrimental to you because in the future as you go through that you're going to start to have issues they're not going to want to participate in the future so make sure that you're including everyone all right so that's kind of you know the big lessons learns that I have from uh the common um uh setting it up you know there you're

you're going to run into other issues too absolutely every organization is different every organization is unique but if you kind of go through what I just talked about and kind of keep those in mind it will absolutely make things a lot smoother so uh common Lessons Learned now so now we're going to start talking about you've gone through the exercise what are the common things that I see that organizations um go through or or the gaps that I see organizations have at the end um in other words everything that you're seeing here I probably shouldn't have this recorded but you these are the common things that we put in our reports for our recommendations so think of this almost as like a

glimpse into you know some of our tabletop exercise reports but the first thing is um IR plan and procedure issues everybody kind of laughed at the beginning when I asked if you had IR plans um hopefully you you really do um but what we've often found that there are definitely a lot of issues with IR plans um the and procedures uh the big thing is that it's missing information the IR plan is missing information um a lot of people like to write that IR plan uh a lot of organizations like to write that IR plan for an audit checkbox and then it gets thrown into a corner and gathers dust that unfortunately cannot happen for an incident response plan so I look

at an incident response plan as a high level flow of the incident uh so in other words um who is doing what you have those defined respons responsibilities what is legal's responsibility what is the line of business responsibility what is Communications responsibility what is ir's responsibility and so on um also have uh defined roles who who is going to be the Incident Commander who's in charge of the incident who is the one who's in charge of the technical side of the incident who's going to be going through and um uh doing uh uh sorry who's going to be going through and uh talking to the help desk to clients and and so on just to make sure that I'm right on time we

go to 12 for this one right all right cool right um also you know you should also in your IR plan you should also have things like priority ratings and slas around them so in other words if you don't have priority ratings the way that I look at it is every incident is the exact same priority so if you have a help Des person click on a fish that's the same priority as a full-blown ransomware attack if you don't have those def find somewhere um now hopefully you have them to find somewhere but once you do also put slas around them you know how fast are you expected to react to a critical uh incident versus a low incident these

are typically defined within the incident response plan uh and there going to be lots of other guidelines in there too I always recommend if you can have a flowchart like a high level flowchart of how the incident should flow because nobody in the middle of a critical incident is going to sit down and read those paragraphs of information having that image is usually key um Additionally you should also have IR procedures or playbooks you probably have these for it I recommend having them for uh incident response as well these are typically a little bit more detailed they may not go to the the level of detail of we you need to contain a system log into this console

click on this button enter this into the field and and so on um it could be a little bit more generic than that where you say something like to contain systems you go to crowd strike and uh quarantine it there you pull the network cable you contact the network team and have them VLAN it off or so you know have all those different ways basically so that you know when you run into that incident you kind of know what to do um you you have that idea list there of all right well plan a is to quarantine it in qu In Crowd strike if that doesn't work plan B is to you know go to the network

team and have them you know unplug it from the network or something like that um so just have those listed out there um I also recommend that they're broken out by incident type um so you have one for ransomware you have one for fishing for denal service and and so on just so uh and a there's going to be a lot of overlap honestly between some of those but um it is going to be a lot easier uh to hand that off to like a junior level person and say hey go do step three of The ransomware Playbook and and do this uh and so on um in terms of how often they're modified uh IR plans are probably

something that you probably you know update once a year however Playbook should be living documents you know do not throw them first off don't put them in the IR plan don't put them as an appendix in the IR plan because you'll never be able to update them uh and second as you get new technology as you get new procedures on how to do things as you find out a cool new way to do something um update your playbook for that uh and and so that's what I mean by by living document and like I said um these are not audit checkbook uh you know checkboxes uh everybody needs to be familiar with these yes roughly how many

pages do you think an IR plan should be that's always hard to say I mean so so the question is how how long should an IR plan be you're right it's it is very dependent on the organization I would say if it's like two pages that's way too too small I I would say you know again depending on the organization um the the good ones that I've seen are usually they they're in a digestible format so probably five to seven pages of content you know that could there could be like images which Bloom it out more and you then you have your appendix which lists out everybody's uh contact information and so on but the the main contact or

content I I mean I think you could probably fit that within five to seven pages um there there are some good templates on online I I don't know where they're at off the top of my head but there are some good templates online

yes so so the question is do do I recommend having one kind of General incident response plan and then the the rest uh the the playbooks kind of breaking out the different types of incidents is that what you're

asking right resp oh okay I got do do we recommend having different in response plans for the different environment because cloud is going to be different than on Prem and you know you may have some other OT and stuff like that um the the general incident response plan I would say probably could still fit into that um you may need to have a couple call outs for like if if it's in the cloud then you know we also need to involve these people and so on playbooks absolutely you should absolutely have stuff specific to that Environ to those environments um the IR plan uh I don't think you need to have separate IR plans cuz again I look at the IR plan as like

this overarching document for for incidents within a uh within an organization it should not be so specific that it names all of your Technologies that's another mistake that I see people make when they're making IR plans is they make it very very specific to their Technologies and what ends up happening is again that IR plan doesn't get touched for like two or three years and when you go back and review it you ask them oh well do you have this like oh no we got rid of that a year and a half ago so um yeah just I I think you know you can make the IR plan kind of generic for everything and then you have

those specific you know other plans or procedures for all those different environments yeah exactly all right uh what's this oh insufficient communication plans that is the uh that is another thing that I see uh organizations have issues with um like I said communication is um very important during an incident especially one that involves a data breach uh a lot of organizations feel like they can control the narrative or they can wait because the attacker isn't going to release information on this cuz they want their money that is absolutely not true especially once you get into the extortion demands for a ransomware attack ransomware attackers are moving we're seeing a trend where they're moving to doing less encryption and more

data breach more data stealing because they're finding that's what people are paying for and I always get asked how how how many of my clients pay uh The Ransom and this is a rough number it's it's we don't keep track of those statistics but what it comes down to is I would say probably about 50% of our clients pay the ransom and of those 50% the majority of them it's not because they can't recover their data because they can they have the backups it's because they don't want that data being leaked um or or put online um but attackers are going to do everything that they can to control the narrative and force you to make those decisions by

contacting your clients contacting your Partners contacting your employees they are going to uh contact the media they are going to put their your um information up on their name and shame page and the thing to remember too and this is important to kind of let Executives know is and and I think the communication team of your organization probably understands this but not everybody might it's not it's it's irrelevant whether what the attackers put on their website is true or not it's more the perception of the public of what's being put up on there um so for example there have been a couple cases in the last you know couple years where supposedly a company has

been compromised and the attackers put it up on their website and the media just swarms at this oh you know so and so got compromised I think the latest one was what was it ring uh within the last like uh couple weeks supposedly got hit by a ransomware attack although I think it's starting to turn out that it's it was wasn't ring it was a supply chain uh person two ring that happened to have some some of their data so ring was never compromised there was you know somebody else the attackers it doesn't matter they want to get their money they're going to put up oh we compromised ring uh and so you know again it comes down to the

perception uh you need to they need to understand that but because of that we see often in tabletop exercises that the uh the communications team or whoever is in charge of communications really doesn't understand the full extent of what they're going to need to do so in an incident uh especially a big incident you're going to need to communicate with uh internal employees clients Partners law enforcement regulators Insurance media in the public as well as probably a bunch of other people uh if you're a public company you're going to need to probably communicate with shareholders um you may have a parent company you need to communicate to them uh if you did I put cyber Insurance on here no if

you have oh yeah I did uh cyber Insurance you definitely need to communicate uh with them and so the organization needs to understand who they're going to communicate with how they're going to communicate with them and when that's the most important part is the when and again it comes back to they are not necessarily going to be able to control the narrative the the company needs to absolutely get ahead of the story when they can but the attackers May force their hand on on this and or more importantly your own employees May force your hand on this there have been many times where uh somebody has leaked information they you know posted a screenshot on Twitter or

Tik Tok or you know some other social media site says hey my company I get the day off my company got hacked and post that screenshot of that ransomware note on their screen it happens all the time uh especially in like I've noticed with like universities this tends to happen a lot because you have the students in there they they don't really care and so they're posting everything so yeah it's uh you need to make sure that whoever is doing your Communications knows what they're going to do have those has those playbooks uh in place uh they need to have their own playbooks on how they're going to communicate to all these different organizations when they're

going to do it and so on um I also recommend it might be hard to see at the bottom there have some templates uh of what you're going to communicate already written up obviously you're going to change that for the situation I would never recommend just having a template and just blindly throwing it out there it needs to be changed for whatever specific situation you're going through but have that language already determined you know kind of kind of know what you're going to say and then modify it for for each situation because that's going to save you a lot of time as well because inevitably what happens when you're in an incident and you need to

draft these communications the communications team or or media relations or whoever it is they draft it then legal has to approve it then leadership has to approve it then probably some other people have to approve it then legal has to prove it again and then Communications needs to make all the changes so uh fix all the changes to make sure it sounds you know uh legitimate and you know it doesn't sound crazy uh and so there's this whole process of going through it it's not you're not going to push out a press release within 20 minutes is what I'm trying to say unless you have these templates that have already been approved along the lines of

communications as as well not just having you know insufficient Communications plans but not having outof band Communications um you need to know what to do when your communication systems go down or you can't trust them that's the big thing everybody in here has email you you probably have some type of real-time Communications like teams or slack or or something like that teams or slack might not go down but you may not be able to trust it you know if an attacker has infiltrated your environment they may be joining your calls they may be joining your incident calls they may be joining your Zoom calls it has happened multiple times so you may need to go out of band when I

say out of band I mean using some type of communication system which is completely segmented from your environment not connected to active directory you don't log in with your active directory credentials it's not on Prem it's could just again completely segmented this becomes very important too for those communication messages you can't necessarily use your main website if you're under a Dos attack or your main website has been compromised you can't use that to push out information to the public to the media and so on you need to have something else already available now some of my uh team members don't agree that you should go this far I I disagree with that but when I say

out of band there are a couple things that I think that you need to kind of keep in mind and and try to set up uh the first is email you know have some way to have out of- band email and and when I say out of- band too for all of these it isn't necessarily your entire organization your entire organization doesn't need an autoband email account your incident response teams and your leadership they do the the people who are responding to this event that need to continue these communications they're the ones who need to have most of this um but email uh realtime chat you know some way to you know especially since we're all working remote or a lot of us

are working remotely now you need to have that way to stay in uh real-time Communications uh voice uh you may not be able to trust your phone systems um you may not be able to trust your cell phones uh and so on and so have some type of Voice network um One Way employee messaging this is something that most organizations have more for Disaster Recovery some way to like blast out a message to all the employees that let's say you have like a snowstorm or a pandemic that you know you telling people don't come in you need to have that one-way messaging system to uh people now the one gotcha I've seen with this though is that a lot of these

systems for organizations are opt in uh so most of the employees don't have that uh have not opted into that so even if you have this one-way uh messaging system to all of your employees now make sure it's going to it actually will go to everybody and not you know the 25% of the organization who has opted in

sing uh I have not seen a one that does all of this at least there might be to be fair I haven't looked too deeply into this um the what what I've seen most often is except for the oneway uh employee messaging everything else you can usually get through like a set up like a separate Google infrastructure so you have Hangouts and Google Drive and Gmail and you can actually control it pretty well there are other you know Services similar to that you know that would be good yes I'm

concate redund and absolutely would so so you know the comment was you know wouldn't have excuse me wouldn't have uh having you know different services for these be have levels of redundancy or create those levels of redundancy it absolutely would but it would also create that level of extra level of complexity um for it and yeah for it is probably not an issue you ever tried to walk a CEO through logging into their email um now now try doing it for a system that they've heard of once a way to combat this though is one one of the things that I've seen done is um set up like Chromebooks uh and they're called the autoban Chromebooks they're already

like preconfigured to connect to everything you just have to enter in your credentials and yeah you're probably going to have to reset their credentials which to be fair I probably wouldn't remember credentials that I set three years ago anyways um but you you have those so they already have something that's separate that they can just turn on and you know start working or at least get the information they need um file storage communication sites and so on uh big thing with all of these is don't wait to set these up set these up before an incident happens there's a really good story I think it was I I think it was Atlanta uh the city of

Atlanta I I could be wrong on that but um uh I think it was the city of Atlanta they got hit with a big ransomware attack a couple years ago they told all their employees go go set up Gmail accounts and we'll use that for the time being the issue with that was Google saw this huge influx of new accounts cre being created from the same location and thought it was a Spam attack uh and so they killed every single one of those accounts uh and now the uh city of Atlanta had no way to communicate um so again go and set these up now don't wait to do that uh like I said set it up now um one

thing too uh for the file storage um upload all of your IR plans and procedures and things like that to it beforehand because you're going to need offline copies if you keep them on a file share in your internal Network and you have a ransomware attack and that file share gets encrypted you you may be out of luck or just print them off you know go old school yes so quick question how do you adise folks people go back and with this on internal Communications and it around yeah it's that's a great question so um you know how what do we advise uh people to do around internal Communications and wrapping it around attorney client privilege um my default

answer is to say go ask your lawyer uh but um what I've seen organizations do is the those types of communications typically aren't under attorney client privilege when you're doing a blast out to the whole organization but they're also very generic there's something I wouldn't tell the organization or tell your helped us to tell people calling in oh we have a ransomware attack you things are down you know we think that data has been taken no just say that you we are having technical issues we're working on them we'll send you an update once it gets uh fixed now of course people are going to talk they're going to figure it out but the official

Communications that somebody is then going to forward to the media or their relative who's a reporter are going to be a bit more generic um so anything that I would say that my recommendation again I'm not a lawyer but um anything that has any information pertinent to the investigation should be wrapped under attorney client privilege but that should also not be going out to a large number of people it should be that kind of tight-knit incident Response Team all right um so this is just a Shameless promotion for me um I wrote a blog post uh a couple months ago about um why out of- band Communications are important this kind of goes over everything I said in the slide if you

want to go out and read that um Jack my numbers up internally for the blog post stuff like that but in any case um another big issue that I see is tools and third-party providers um a lot of times people think or organizations feel or think that their tools are going to do something and when we get into an exercise especially those Hands-On exercises they don't or there are those edge cases where they don't work like you expect them to and that's what you know to be fair that's what uh tabletop exercises are for to help figure this out but um what I see is you know a lot of times you know tools either don't

work as uh you expect them to the example I gave before you know how do you how do you obtain data from a system that's been quarantined uh I've had issues with that with organizations who have ironically one organization that I'm thinking of they're they've been a client of mine forever done multiple tabletop exercises with them swore up and down in the tabletop exercises we would quarantine the system we would ask all right well can you get us data from that this is what we would ask yes we can absolutely get that had an incident they quarantined the system we said all right well great go get us that data well how do we do that like you literally told me

five times that you could do this um go through and you know that again that's kind of what those Hands-On exercises are for it helps kind of great you're telling me this now prove it um you know kind of that trust but verify uh type deal but you know again make sure that your tools are working as as as expected um one of the big things that we've also seen is uh organizations tend to have a lack of coverage on uh their systems this comes out in an exercise where uh yeah they have uh VR but it's only deployed to the workstations it's not deployed to all the servers or it's not deployed into OT or o there's or there's

zero visibility into OT and yet we're doing an OT tabletop exercise um so make sure that you have all the that visibility that you can yes what's EDR and OT oh yeah sorry so EDR is um I'm going to get This Acronym wrong but uh endpoint detection and response think of it as like yeah antivirus on steroids where it's instead of like it probably has some signatures in there but instead of looking at just signatures it's also looking at the behavior of the system you can t typically do a little bit more of um uh actions on it so you can sometimes do some forensics you can grab data from it and so on um OT is

operational technology so think of this as like manufacturing networks uh power plants the the the stuff that's kind of controlling our lights and HVAC and and stuff like that basically the the systems that are running Windows 2000 still and and you laugh but I literally had a case 3 weeks ago where I had to figure out how to grab grab data off of a Windows 2000 system yeah it's uh but you still need to have that visibility into those environments and that's one of the big things that comes out of tabletops um dos attacks um that is a huge issue um do you see them all the time no but it does happen and for a while with

ransomware attacks attackers Were Us utilizing deos for uh to to kind of extort you more and one of the things that people don't think of until it happens with DOs is that um you can't use your internet so that means you can't use your cloud system so all that like stuff you have in the cloud all the like consoles for your EDR that's in the cloud you can't use because your clients cannot connect to it um so kind of you kind of have to think through through some of those things and finally validating providers will do what you think they will um I know we're kind of in the last 15 minutes here we're almost

done but I wanted to you know tell a quick story I did a tabletop exercise a while ago where um we had I I so I always invite um or recommend to have third-party providers on uh tabletops that's probably one of the people I should have put to invite um because if you're going to rely on a third party service to help you in an incident you know absolutely bring them in ask them what they're going to do you know find out what their procedures are and a client had done this they had brought in their um their mssp their their managed security provider and and we're going through the incident I think it was like

a ransomware incident or something like that and we get to a point where we're asking all right well who's going to investigate the the the incident and the client says oh well our mssp is so I started asking questions and the mssp was kind of dancing around the answers until I finally just honestly just got sick of you know somebody dancing around my my questions and said all right here's the situation we have this disc it's been taken offline it is patient zero it needs to be forensically analyzed are you going to do that and they kind of stopped for a second and then said no that's outside the scope of our responsibilities and the client did

not know this the client assumed that they were the ones who are going to be doing this for this so a huge uh Gap that came out of this exercise was that they now needed to find an IR retainer that would do this for them um so again this is a great way exercises are a great way to kind of get those thirdparty providers to uh you know validate what you think that they're going to do for you and not only that what what's the time frame that they're going to do it for you what are their slas um I will say if you do have an instant response retainer and I'm saying this as an instant response retainer ask

them what their current workload is or what their slas are for getting uh for like responding and starting to work not only that ask them what do you do when you have a surge because every incident response uh company they have to figure that out we have I I don't even know how many retainers we have um but actually you know I'll give you a little bit of insight scoop here um the one of the ways that incident response retainers figure out how many people they need is they look at the number of retainers and then they create a ratio of the number of their Consultants their their analysts who can do inant response to the retainers and

it's usually somewhere between 8 to 10 to one so for every uh 8 to 10 retainers you have one person who can work on it um normally you're not getting every one of your retainers calling you think of this as like uh you know Banks you know Banks you know have you know with Silicon Valley is a great example banks have all this money in there but they they really don't have all the money in there um and so if everybody everybody does a run on the bank then the bank is in trouble if every one of my retainers called me I'm in trouble um because I don't have enough people to work all of

those incidents and so you have what you call surge uh procedures you know ask them what those are uh you know a great example of this was log forj everybody got called for log forj half neum everybody got called for half neum and so you you have these attacks we have precedent for these attacks that they need to know how to handle and they need to be able to service you because that's within your contract so so definitely ask them about this um we talked about Cloud uh so a lot there is definitely a lot of lack of cloud capabilities that we find within an incident response um so there's a lack of monitoring response containment

restoration training vendor issue there's lots of vendor issues what it comes down to is um and I tried to make the the images a little bit you know to what I'm talking about the the cloud is definitely the wild west right now um especially in incident response uh incident response is one of those things where honestly you know cloud is is a different Beast is what it comes down to and so you need to have special training and organizations need to understand this and when we get into tabletop exercises cloud is just one of those like magical areas that they just assume will be okay and it's really not and we we are seeing more and more uh attacks

in the cloud not just systems in the cloud but also the cloud consoles attackers getting into that if you have a um screw up the acronym but platform as a service is that right where where you have like a like a Heroku type thing um you know attackers are definitely getting into those um and so you know you still need to uh Salesforce is a great example uh you need to have you know response capabilities into that and and include those in and absolutely include those into your tabletop exercises one I just did was we it was a breach within Salesforce and everybody swore up and down oh nobody could ever get into this good thing was my contact

at that company had actually done some research for me and found that yes you could actually breach our Salesforce uh uh Dev environment which just so happened to have production data into it and everybody in the exercise said no no you can't do that that's absolutely not possible and and so we said all right well hold on he pulled up a movie and showed exactly how he did this and the entire call got silent it was actually kind of awesome but um the point and actually this leads great into my next Slide the point wasn't to like shame everybody it was just to show that yes this could actually happen you know one of the things I didn't say when you're

designing your scenarios make them realistic don't just say oh we have ransomware you know magically ransomware has appeared in our environment what do we do I mean yeah you could go that route but it's not realistic plan everything out you know how did the attackers get in all right they got in through fishing great How would how would somebody bypass our fishing uh defenses you know and you this is a great way to kind of call out those little gaps that you've been trying to fix forever put them into a exercise and then it becomes a gap within that um exercise um all right I'm going to go through I'm running out of time so I'm

going to go through these a little bit quickly um insufficient containment um the big thing oops sorry um the big thing here is that a lot of organizations they have a way to quarantine or contain systems like one or two systems or maybe a dozen what are you going to do if you have to quarantine 500 systems at once that's where a big gap lies for a lot of organizations so that usually revolves around you know needing to script it out to to work with your API of whatever you're using to quarantine systems um make sure that you can do that or at least you're thinking about that uh also make sure all environments are covered

again this comes down to making sure cloud is covered making sure OT is covered making sure the executive environment is covered just everything I've even had uh I've even seen organizations go so far as to make sure that their employees um mean not their employees but the executive homes are covered as well um over Reliance on one person we all have that person in the organization who kind of knows everything they they've been in there forever um don't rely on that person um they are going to be gone during an incident it happens all the time or if they're not gone in an incident what if you have two incidents at the same time it does

happen you may have three incidents at the same time they can't work all three at the same time so what this comes down to is if you see this within your organization make sure that you're spreading the knowledge make sure that you have more than one you there's a reason we have backups um and also document everything that you can this goes back to having those IR plans and procedures make sure that everything there is documented um talked a little bit about this already um extortion discussions uh especially on the executive side this is what Ransom payments the data breach payments things like that uh thing to keep in mind is um and make sure your

Executives know this they may not be asked specifically to pay a ransom you may have your data up for auction and you may need to buy that data to either know the full extent of what the data breach was or to prevent somebody else from getting that data you may need to make a payment decision because of that uh and so there are a lot of questions that your Executives or your leadership need to um be able to uh make when or know about when they're making these decisions like uh under what criteria would they pay who negotiates it who's going to facilitate it what if the ACT actor is sanctioned um the big thing on

the uh facilitation is a lot of organizations think that insurance is going to pay their U ransom for them it is not I mean check policy it might but chances are it's not going to it's more of a reimbursement thing up to a certain amount uh and so it becomes an issue there

yes yep yes so so the question is or the comment was that you know are are you supposed to contact your cyber insurance policy early on absolutely you absolutely need to and in fact that's one of the the the discussions you should have with them especially on these exercises if you can get them on an exercise um is find out when are we supposed to contact you because one of the things that insurance companies might do I'm not saying this will happen but one of the things that they could do is if you don't contact them at the right time they may just say no you didn't contact us so we're not going to pay anything um I haven't seen

personally seen that happen I wouldn't be surprised if it could happen but yes absolutely you know make sure that you understand you know especially with payments who's going to doing that will Insurance you know negotiate for us and and so on big thing here and I know this is probably obvious but don't document when you would pay because the attackers will 100% find that document and then meet every one of those criteria and then they're going to throw that document back in your face um so don't document that um document more of who makes the decision and then have those offline discussions I think this is my last one is it yes this is my last one um so uh

don't go into an exercise having bad int intentions unfortunately I have been in those exercises where it was the goal of somebody to basically name and shame of other groups don't do this please first off for the moderator it's a horrible experience to go through um but second it it's it's not beneficial to the organization yes you're going to have disagreements yes you're going to find gaps but don't don't have this turn into a finger pointing situation exercises are meant to find those gaps so you can make the organization better and move on from there um so like I said don't shame other groups don't bring up past issues uh especially if they've been resolved

like oh well you know this group failed in in the past you know we can't trust them or you know anything like that don't bring those up um don't call people out directly uh don't argue about every little detail now granted um it is uh part of the moderator's responsibility to make sure that you know things move smoothly um but you know again it does come down to the participants and like I said before I have had to almost break up fights in uh in exercises yes this maybe obvious but maybe on the fifth but don't think you can't learn anything you know there's some people that feel like you can't teach them anything exactly you got

everything covered your waste you might on yeah and and I think especially on the business side you know they they probably don't understand or they don't know what they don't know um and so yes everybody I even myself who has I I've run Man in The Last 5 Years I probably run at least like 2 to 300 tabletop exercises I always learn new things every organization is different I every once in a while I'll have an organization throw something at me that I I it's brand new to me I have no clue it's like things like vdi and stuff like that you you never think about those until the very end or until something happens and so yeah you absolutely can

can learn something yes what's your opinion or I guess experience with Ransom payments and how Banks May react to that when you're trying to purchase Bitcoin during um so the question was you know how have I seen uh rans of payments go especially like with banks fortunately I'm not involved in that process um I don't want to be involved in that process um my my job is pretty scoped down um that being said I know that organizations that I've talked to have had issues with buying things like cryptocurrency um depending on their vertical depending on their organization and so on so one of the things that we've had some organizations do recently is kind of come up with those of how

would we buy cryptocurrency or maybe even having like a cryptocurrency reserve now that's these are huge organizations that do that um I don't I wouldn't expect like a small to medium business to have like a cryptocurrency reserve but um you know at least have those ideas or you know more importantly talk to your insurance company to see if they'll do it some IR providers will facilitate that transaction for you talk to them um or they at least know somebody who does like I said we don't do it but we we have partners that do uh so you know there's it kind of comes down to you know going out and asking um almost out of time think I have like

a minute and a half left again the big thing with tabletop exercises is you want it to kind of show those gaps and you want it to end like this you want everybody you know not everybody's going to be celebrating and happy at the end of a TBL exercise but it should not end like this um so uh yeah um hopefully you know everything that I that I talked through here um uh uh all all the mistakes that I've seen in the many years that I've been running exercises will hopefully help uh some of you here if uh I didn't cover something or I went through something quickly especially at the end there I know I'm

out of time right now um please feel free to contact me I'll be at the conference all day uh feel free to email me I'm always happy to answer questions and so on but I'm out of time so thank you very much I'll be outside to ask answer [Applause] [Music] [Applause] questions be interested that