The Crushing Chaos of Corporate Crisis: Bringing Our Best to Incident Response

I would like first and foremost before anything else running a conference organizing a conference setting up a conference is an incredibly large amount of work please give harvinder and the rest of our volunteers a really big round of applause thank you my [Applause] friends what rushan chos of perp crisis I like alliteration uh inant response has been a pass of mine for a very long time it was actually a a really weird space in my life where I said wow I'm really good in a crisis and then I had to have that introspection as well and say crap do I just create a crisis everywhere I go because I'm good in a crisis of course

we don't want that either uh so I have a few characters that will come up on screen I try things differently sometimes at bsides and I love this audience I love these people because you're you're willing to experiment as well and do things and see things that are a little bit different because I spend my entire workday 40 hours a week in PowerPoint hell and sometimes we can do something a little bit different so that's what I thought I'd try uh over the last many years since I left High School in like the N early 1990s oh my age um a lot has changed a lot has you know really evolved and there's nothing in

our businesses that we work in CU we all work for a living I'm assuming most of us work for a living we're going to school to go to work for a living it's part of the life that we have is we end up in technology crisis because it's technology people no matter where we land in the stack at some point somebody says Dakota I know you know the I am system so um we're pretty sure that we're under a credential stuffing attack so I needed to solve this I know it's 4:30 on a Friday but uh bye good luck with that uh but there's there's a broader implication to everything that we do in technology so I was just trying to have

a little bit of a broader look at all the ways that this affects us and all the ways that this affects everybody in our business so that's part of why I I'm just I'm messing with the mic cuz our sound man is awesome look he oh he's right on it look at him go that's great uh we we have different ways that we interact there's everything in our business runs on technology now so how do we help organizations navigate it when the big one happens so when I'm talking about instant response and corporate crisis I'm talking about bigger than Sev one we all get those people in our businesses who say Sev one we got a Sev one everybody all hands on

deck at S one and then you find out that actually it really isn't all that big of a deal and you solve it before you go home at 4:45 on a Friday or there's the big one which involves ransomware or privacy breach reaches or in what we were just listening with Michelle's operational techn hi Michelle hi hey good to see you you uh so we have failures in operational technology that can have life safety implications we can have privacy breaches depending on what the Privacy breach is that can have life safety implications so there's a whole lot more to the work we do in incident response and technology in general that that really can be stressful on people

that can have big impacts and when we're dealing with some of this stuff too we're sometimes having to take people out of their regular operational mode and ask them to do different things I think that's where I started yesterday you said we don't need that server online today you say we need that server online before we could move to the next step I can't work under these conditions you keep moving the goalposts everybody anybody ever had that has anybody ever said that CU they're not wrong right that is the classic it guy who's been pushed some stuff these are the instructions you have for tomorrow this is what we need done uh I think it was dragos who coined

the the the do now do next do never and when you start telling people do never they cross stuff off the list I'm not saying that that that the mentality is wrong but if you tell someone it's not important we don't need to worry about that or you give them any kind of indication that we never have to worry about that ever then that gets knocked off the list it's a real real issue because there are a whole lot of people out there in our organizations when we're dealing with uh when we're dealing with crisis I'm wired for this stuff right the the the cyber security equivalent of living life a quarter mile at a time but a lot of

people aren't they have other things going on in their lives other things they want to do other things that are already stressing them out and we're asking them to act with speed plain language is really important as well because not everybody understands all of our acronyms all of our you know special ways that we talk about things and threats and vulnerabilities and exposures and all that stuff plain language really helps another thing that I found that works really well is figuring out what language especially the information technologists use if they're if you're a agile can band shop push stuff to backlog yes we have very important things we have to do right now we have

very important things we're going to stack up everything else is backlog and information technology professionals who understand uh agile will get that

naturally so that's item number one all of these quotes that I have in here I got eight of them that was number one these are all things that have been tossed at me over the course of my career and instant response to and some of them have really hit me that one hit me pretty hard because I was like what do you mean you weren't expecting to turn that server back on and the answer was because you told me it wasn't required oh crap so I had to do a lot of backpedaling there and that's what this is about what we're working on now what we're working on next we'll re-evaluate in the future because the reality is is we got

stuff we got to get done we're moving at velocity we're not in Normal operations mode stress is handled better by some than others as well this one was another one that hit me kind of hard this is one actually goes back about 15 years oh hey uh our team lead was just taken to the hospital by ambulance crushing chest pain really weird pulse just wondering who's going to lead our incident check-in call tomorrow morning you notice that this prick also did an offer to do it himself when I recorded these so I I I actually brought back into my mind the people who had thrown some of these things at me over the years and my

middle kid helped me pick out some some clothes and stuff the original one that I had for that I recorded and then I watched them again and I was like wow that is such a punchable face oh so I had to back off a little bit and and I actually had to re-record those cuz it it gave me a little bit of just horrible ideas of of all those things so where that quote came from and it's happened at least once more in my career in incident response and doing crisis response is people who have actually snapped and it's not a badge of honor it's nothing that we have that we should be proud of that the instant response

work we do is hurting people in the case of of that quote right there that individual who ended up having to go to the hospital did have a major heart attack ended up with a triple bypass um had all sorts of horrible other things going on in their life they hadn't exercised in a couple of decades and had a whole lot of family stress too but that's the reality of the work we do some people gravitate towards stress some people don't and that's something that I have I have become a little more uh a little more angry about is when people come at me and say things like they knew what they were getting into when they signed up in this career

career I thought I was getting in for 40 hours a week worth of monetary gain for my family not you know potentially needing an ambulance ride so the we don't know what's going on in other people's lives especially when we Jam them into crisis response with a million do Nows and a million do nexts and all this pressure to say well sorry it's all on you Logan oh by the way I'm not making our 3:30 meeting yeah sorry but we're here so as far as I'm concerned we're good um but we're putting a lot of pressure on people in these things and it can snap them I've often put as well into instant response plans when I have the opportunity to

review an instant response plan with a company one of the things that I always make sure is in there is a link or some sort of mention of the company's benefits plan most companies have a crisis hotline something you should be looking up yourselves is what does my benefits provider give me if if I need some help and don't be afraid to use it the bravado that comes with crisis response and information security instant response whatever we're we're calling what we do really needs to evolve a little bit into we take care of each other along the path there's no value in broken instant response superheroes your dedication to your company will not get engraved onto your

Tombstone and I'm not advocating you know just bail at the first sign of trouble but start doing things in your in your in your work um what I made notes take breaks when you're in your Prime not until you've long passed it succession planning if you are the only person that can do your job you're already in trouble be less judgy I actually have less judgy is one of my notes be less judgy the people around you uh incident I worked on in it was it was a very very messy very large ransomware incident in in March and there was some judg going on when people were saying yeah you got me until it's it's it's Sunday my

daughter's fifth birthday is at 2 pm. you've got me till 1:30 and people were go really really he's leaving for his daughter's fifth birthday yeah absolutely and if there is no succession planning available then you know that's that's another failure there's often a bigger picture in a lot of this stuff that sometimes we get our blinders on um and this next one is is all about that was me having my blinders on and getting something tossed at me that was a little bit freaky the attacker was Pinchy spider Pinchy spider has been in our stuff for eight weeks and you're only advice now is rebuild everything buy a whole bunch of new laptops get a new Cloud tenant fill the

Legacy data center with coolant water from Chernobyl reactor 4 here's the deal we have 3,500 employees who can't access anything right now we're burning through $2 million in operating costs every day with nothing to show for it we're already in a hole we might never climb out of rebuild everything what else have you got I have been on both sides of that conversation where yeah my answer has been you have no effective backups you have an attacker who is an advanced persistent threat who's been in your environment you need the Scorch to Earth approach you need to wipe everything start from scratch all new all new accounts all new everything and the response coming back of you know um

that's not in the cards right in a couple of recent ones that's been you the the security team has come in and said we can't give you any indication whatsoever that you can ever clean up from this you cannot clean up there there is you're asking for a guarantee that you're going to be clean at the end of this we cannot give it to you rebuild everything if that's what you want and that's where this one came from was uh no not in the cards there was stuff that we were finding in that environment that was fantastic anyone familiar with an rs6000 right like that's that's the those are the big fridges that were in

Vogue when I was in high school in the 1990s and they're still running you cannot turn that thing off because it's not just to flick the switch turn it off turn it back on to get that system back on you need specialized resources who don't exist in the market anymore or maybe there's one or two specialized people out there but good luck find ing them talk about uh succession planning oh other ones that uh call every customer you have to make a phone call to every customer that one was great right no we there's there's 14,000 customers and you want us to make 14,000 phone calls I don't know that we're going to do that

just some of the the the demands that as Information Security Professionals sometimes we push out to our clients to our organizations that we work for maybe aren't uh maybe aren't effective Michelle talked about this you had this up on the slide empathy empathy is incredibly important in information security I remember being at a client session years and years and years ago where there were about 80 people in the room and this was the entire Operations Division of this company and the president of the company is standing up and he goes okay everybody I got my whiteboard I got my whiteboard right here I want to hear from you what does this company do and 80 people working in

operations couldn't come up with a s they uh we sell things okay what kind of things do we sell and he was I could see him going through the five stages of grief like denial okay we're going to get some really good answers up here and then he went to kind of come on guys we're got to get some good answers here it's like okay if anybody comes up with a good answer I'll give you five bucks right bargain in and all the finally he got to acceptance and he quickly moved on to another topic but there the people who run these businesses we work for are typically pretty smart people but they don't get us just as much as we don't

get them very often we don't understand what their motivations are so that's where I often recommend to people ask questions that are above your pay grade I see that in large organizations and government organizations all the time people that above my pay grade I'm not asking that question if you want to be successful in instant response in crisis when stuff is going bad ask questions that are far beyond what you should be usually privy to and you'll find that a whole lot of people will unload on you it's great I love it I'm not saying ever don't the note I have here don't accept don't accept uh uh accountability for bad decisions because sometimes that's what the

Business Leaders will bring down to us is say well I'm going to do this you're going to do this this is the plan do this and sometimes we have to make that choice I've had to make that choice a couple times where I'm saying okay let's just get that in an email that we have recommended something and you're recommending something else sometimes I have never expected to get it in writing and I've covered my butt simply by sending the email so that I can say you were notified on such and such date that this was a bad idea and you did it anyway and I told you so never feel good okay they actually sometimes feel really

good but that covers our butts so we're not taking accountability for their bad risk decisions what we're doing is making sure that they are making risk informed decisions sometimes the risk tolerance of a business owner who is in crisis is going to be very different than our risk tolerance as as technology and cyber Security Professionals cuz well oh and cuvies I think it's number five of The Seven Habits of effective leaders I try to remember the book it was a very long time ago seek first to understand then to be understood I think that we are all very smart people if you're here you're a smart I believe that you are all smart people especially

you Jeff we're all very smart people so sometimes especially in crisis we walk in as the smart person is arrived the smart person is here everybody should listen to me but we have our own blinders on and we're not going first and foremost to say hey uh am I right in assuming that this is more important than this seek first to understand then to be understood at no point does it ever actually say that we have to agree and quite frankly we're unpopular enough as it

is how how's it going it's never a good time when we're on three calls a day with you Adam I was the one that I wanted to punch that was I have had you know as as someone who comes into organizations or have worked with the in organizations where we've done crisis response that's one that I get fairly often because people are in crisis and they don't get that we're there to help they just know that uh something bad has happened and this person is connected to the bad things happening so I'm going to put all of my Venom on Adam okay right it's something we're going to have to accept something we're going to

have to take sometimes and that's okay the reality is is we definitely have to be very mindful of the energy we want to bring with us to high stress environments that particular one that was uh that was actually the first time that I found that to be venomous cuz usually when someone's tossing something like that at you it's some kind of you know joke right it's someone's attempt at being funny and sometimes it is funny sometimes it's very entertaining sometimes it's really from a a place of fear that you've got somebody that you're working with in a fear Spike who sees your face on the team's call or in the boardroom and just you being there

as the crisis Handler as the instant responder as the person who is is trying your best to solve these problems is sending their blood pressure through the roof and couple things that we can do to get better at that uh yeah deciding what kind of energy we want to bring to the do we want to be energy positive do we want to be someone who is a booster someone who is helping people really come to terms with what's going on and what's required of them do we want to be an energy vampire I hope not we all have those people in our lives too who are just sucking energy from our lives and we don't necessarily want to be that do

we want to be someone who is seen regardless of what we're doing in crisis response do we want to be seen as someone who can handle things or do we end up being seen as someone who maybe requires handling and it depends on the situation sometimes I'm both I think sometimes we all are something that can help us with introspection on that a little bit of an out outof Body Experience something that I've I've coached a couple people in doing that they've used has been helpful you know team calls are great for this because if you got somebody who's using a headset they can record their performance with no fear if all they're getting is one side of of

what they're doing record yourself and listen to yourself back I watch these videos and believe me I'm cringing all the way through it's brutal but record yourself play it back find out how your energy is coming across do that outof body experience to try and figure out who you are what you are and what you're doing um something that I found is really helpful especially with people like that one greet everyone like you're greeting an old friend harender my brother right greet everyone like you're greeting an old friend and you'll find that those people will come back to you with that toxicity gone and some of their fear opening with vulnerability and that's an amazing

thing too because then you can get real performance out of everybody you can get real goals reached by everybody altogether um gratitude approaching things with gratitude and and really saying harender you know this was a great great great bsides probably the best organized bsides I've ever seen in my entire life even on my way here I hit construction out of my hotel and then I hit a wall of traffic on uh on the yellow head and then I was like instead of going oh crap it took me 45 minutes to take what should have been an 18-minute Drive hey I got here and I didn't get lost thank you Google Maps right little bit of gratitude definitely

changes our own mindset to how we approach these things but things like uh greeting everyone like an old friend the authenticity has to be there gratitude the authenticity has to be there if you're Faking It people will know so do what's authentic in there that's all been really philosophical stuff oh you wanted that patient zero and patient one laptops for a full forensic analysis nobody told me that I erased them and I put them back into production so I felt really bad for that it guy because the CIO came after him and said what the hell were you thinking nobody told them that those they went scorched Earth and they needed to come up with 400 laptops tomorrow here were a

couple of laptops sitting on a shelf had their service tags on them so they put them into production they wiped them and put them into production it happens then they tried coming after me and saying hey Adam how the hell did this happen and I was like where was it written down that they shouldn't have been put back into production and it wasn't uh other ones that have happened in in cases like this where stuff isn't written down uh backups this is a very very common one where people say I got 30 days worth of backups and we're on Day 36 of an instant response and somebody says Oo we need those three machines from 30 days ago well I can't

get them anymore because we overwrote them with ransomed data hooray so things like yeah pause your backups if all your data's crap anyways um in one case there was a network device that got sent a power off command that could only be reached by ice roads and it was August so the devices weren't being turned back on ever again well okay it was about 6 months later don't wait to write down your responsibilities all of us are responsible for things in our lives all of us right whether that's our personal lives our professional lives whatever we are accountable for things so if there are things that you know you are going to be held responsible for in

crisis write them down ahead of time we all have you know one note or Google keep paper and Pen doesn't matter but if we can start just making these little notes about the things that we're going to need to pull out of our back pocket in crisis when your stress level is up here you're not thinking clearly enough so if you're thinking oh I'll have that I'll get that no problem chances are it actually could be a problem um Atul G's the checklist Manifesto is one of my favorite books because it talks about actually doing those sorts of things what are the seven things that I need to just simply remind myself of that will keep the lights on

or put the lights back on and there's I don't know there's no cop that'll come and arrest you for a little bit of pre-incident planning uh if you're if you're not responsible for instant response planning I don't recommend telling anyone you're even doing this because then they'll make you responsible for it and then you lose all your hair it's brutal uh another one oh this one this one was good too what do you mean the bill's already $25,000 I haven't even authorized any work yet I've seen this a number of times too whether it's a contractor who comes in and the the security team going wait till the client sees the bill on this one aren't we valuable now if the client

is going to be surprised by a bill then we've already done something wrong ahead of time right or in some cases there were there was an investigations team that I was working with where they were poking into Data with people that they had no business poking into Data with that ended up getting a whole bunch of people in trouble so asking those questions ahead of time what am I authorized to do see what is this going to cost in the case of of of contractor Le or or or client and and and service provider interactions it can be something as simple as I think this is going to be 20 hours worth of work to

look at do you want to authorize that or if you're uh in a employment environment where you ask the question uh I need to drop everything for two days and work on that is that okay find that person who's accountable for those sorts of things and and start asking those questions we all want to go and poke at Pinchy spider and Pinchy spider and doing our malware analysis and discovering what all the lateral movement was and attribution and stuff and we tend to forget the unsexy things like let's actually make sure that between detection and Analysis we've got authorization to actually go do some things um that will save relationships it'll save jobs uh and one of the things that is

really important too is it really helps you understand first and foremost something that is a big mess is understanding the problem we're actually trying to solve first what's the problem and when did it start I'd been on the callbridge for about 72 hours but I don't think I could just describe the problem other than everything

is so th this one is very very relevant because when things are going really bad sometimes it's incredibly difficult to describe why things are going bad to prioritize what's going bad to figure out you know and don't get me I hate big call Bridges that's some leftover from some anachronistic process of we'll get all the smart people in the room and the smart people will go and solve all the problems but the smart people aren't getting any direction to actually go and do anything all they're doing is assuming other people are doing things it's like going to going to the pool party every we're having a party in someone's backyard and there's a pool everyone's watching the kids in the pool

which means no one's watching the kids in the pool that's what those big call Bridges often end up in it's a whole lot of people waiting for someone to do something the idea of a problem statement so I I started writing this into uh playbooks instant response playbooks and people were questioning me originally as like no we want a Playbook on how to handle rant someware I'm like okay we'll write that too but wouldn't it be great if your Junior people people who are not used to doing this kind of work also had a little one-pager that said to describe the problem we're trying to solve we want to do these steps that our business objectives that

we serve in this organization are this it's these four basic things and if there's a problem we're going to uncover it by doing this this and this and then we can describe the problem often when you're dealing with a major business crisis it's not one problem it's going to be several okay you can do that all together and then you bring them together and you you call them a common operating picture because if you've really got the big one what you're going to have out of that is a whole bunch of people who aren't information technology or cyber Security Professionals you're going to have lawyers you're going to have privacy Specialists you're going to have Business Leaders and you have to

explain to them in their words what the hell is going on with the technology what is going on with their customers what is threatening their ability to do business so that becomes really important and simple formula that I pulled out of Emergency Services I think I've talked about it at every bsides Evon the can report conditions actions needs just a quick formula for you to say here's what I know about what's going on here's what we're currently doing and here's what I need from you the audience whether that's a decision whether that's budget whether that's approvals whether that's more technologists conditions actions needs it really really does work it really you should be able to deliver

that one of the in teams where we've gotten good with this we can go around a room with 20 people in it they all get 90 seconds to deliver their sit WP their situation report conditions actions needs 90 seconds each we go around the room we get everything done it it works really well uh oh I like this one my top three priorities you want my top three priorities out you donkey I had 50 oh I hate that guy so five stages of grief I would say that that person is probably still sitting at denial he says he's got 50 priorities chances are he's not doing any of them but it gets worse eventually you're asking for my

top three priorities my top three priorities Adam you donkey I have 50 oh we've elevated to anger we got an angry one there yeah and crisis response gets people angry and what gets people more Angry than crisis response is having too many priorities having too many things that are kicking their butt stressing them out oh God he's asking for my top three priorities top three priorities Adam you donkey I have 50 and that one I think is an acceptance knowing that none of those 50 priorities are going to get done or his 50 priorities will be superseded by somebody else's priorities talking about badge of honor and instant response is tough and you got to be tough to do this

job I hate that crap what is um more difficult for me is is this is when people come back to me and they say Adam I got 50 priorities I can't just tell you what my top three are the first one I even talked about do now do next right do now just get out of bed Adam do next brush your teeth right we have to find a way to if if we want to keep moving forward if we want to do nothing for 72 hours on a call Bridge hey none of this is worthwhile if we actually want to keep moving forward we have to rep prioritize and there's some relatively simple ways to do this if everything's a priority

nothing is right two questions there's only two questions that I feel are necessary to help us get there what's important now and what's in my control and I I navigated the pandemic with a family of high energy teenagers using this too right navigate lots of Crisis using this if it's not in your control if you cannot steer the outcome then why are you expending your limited energy on that objective if you've got 50 priorities and you can't do anything about the outcome of 45 of them well that's 45 that you can put on backlog and what's important now I guarantee that if you looked at whatever's left you will find that there are a small number of things that really

have to happen in order because if something's not important now but it feels like it should be chances are the measurement of success for that outcome will change five times before it actually becomes important what's important now and what's in my control what this also helps with his PO incident analysis the the world of highdefinition video and 4K resolution and super high speed I'm going to go catch the football oh oh his foot was way you could see his foot was like he's going to go catch the football and his foot is just a millimeter out but you can see it in high resolution High defin television that kind of mindset has permeated into everything we do Fort McMurray wildfires

and everybody was applauding and high-fiving the fire chief and director of Emergency Management when that happened then all the armchair quarterbacks came out and said wow that guy was a muppet and he was just doing the best he could with the information he had and the resources at his disp disposal he was doing the best he could with what was important now and what was in his control we're going to see the same thing we're seeing it already in Northwest Territories you've evacuated Yellow knife and you're not letting people back and while you're a bunch of jerks they're doing the best with what they know the information they have with what's important now and what's in their

control so for really really big crisis it also works for really little crisis I was at the airport on a flight to Winnipeg and the flight got cancelled not delayed it was just canell but to one of the people working in my organization it was like Adam it's a must be you must be in Winnipeg tomorrow morning and my answer was there are no flights that I that can get me to Winnipeg for tomorrow morning do you want me to an expense a cab do you want me to an expense a limo no okay well then we understand what important really is and apparently it was not $155,000 for a ride to Winnipeg that would have

been awesome I would have gotten sleep I would have drank a bottle of wine it would have been fantastic uh crashes that happen on Highway 2 in the middle of winter these multi-car pileups what's important now what's in my control safety I have my family in my car I'm going to go find a nice safe spot up in the other side of the ditch and what's not in my control anybody else El's safety I've you know for some reason car crashes you know when even when I'm a pedestrian they gravitate towards me and the last one that happened right in front of my house it was hilarious I'm like dude can you please stop standing in the middle of

the road I know you just crashed your car into a parked car but there's no reason for you to be standing in the middle of the road he was still out of my control but focusing on what's in our control what's important now that butt covering that comes at the end that is something that is really helpful for for all of us too is to be able to go back and justify why did you make that decision Adam it was something that we could control and it was immediately important that's why we did what we did and when we're having our own the cortisol is gone the adrenaline is gone sometimes it becomes really difficult to release that

stress in our own lives and you sit awake at night and ask that question did I do everything I could have done did I do it right was there a different decision I could have made that would have made things go differently and sometimes the answer is no sometimes the answer is yes and we try to learn from those uh so if you're if you're predisposed to depression things like this can help you live with the decisions you've made and just keep moving forward because there's so much that really is out of our control priorities Health empathy energy butt covering for all of you that are here I really do appreciate you I appreciate you you all I really genuinely hope that

if you find yourselves in some crisis event that you will find a way to navigate at the best way you can um I don't know if we want to do questions or anything or if we just want to get the heck out of here is there anyone with anything burning on their mind Michelle wants to Heckle I know we've left her speechless yeah well my name is Adam I really appreciate you all I believe you will all be awesome in your next business crisis because it's just a matter of when not if thank you all very much welcome to bsides [Applause] Evon