BSides Scotland 2019: Closing Keynote - Chris Sutherland

all right so good afternoon everybody I know your butt sore sore your brains are full and most of your thinking please hurry up because we want to go to the pub totally appreciate that so let's start our story our story tonight begins at a lovely autumn day in New York or New York City if you're a tourist or New York if you live down so at 8:46 a.m. on September 11th 2001 American Airlines flight 11 crashed into world trout World Trade Center tower number one about 20 minutes later America and United Airlines flight 175 crashed into tower number two and right away there are about two thousand six hundred and six people killed over six thousand people

injured and in terms of the fire service there are 343 firefighters from the FDNY were also killed that day fast forward about half an hour later at Arlington Virginia 937 am American Airlines flight 77 hit the west side of the Pentagon nice little photo from that here not one there were only a hundred and eighty five people killed there should have been a lot more but fortunately there were some renovations going on at that point in time they're actually reinforcing the structure making it basically more bomb-proof and also because you have like very aggressive type-a personalities or very disciplined individuals they managed to evacuate and take care of each other very quickly and so that actually

reduced the casualties of September 11th so why are we talking about this at a cybersecurity conference I want to talk a little bit today about the fire service responses in New York and a Washington DC the reason is quite simple we can learn a lot from the fire service on how to manage cyber incidents in our own industry but before we go there as a good speaker I get to do a little family self-promoting my day job I'm c74 ing I'm responsible for fraud and cyber at a large global bank I'm happy to say I'm a CTF winner from Def Con from a long time ago but more and more importantly I used to moonlight

as a firefighter in the Northern Virginia area that our department actually was one of the departments that responded to the Pentagon attack in September 11th i'ma Fire Service I was known for my work ethic a very dedicated hard-working person you know great leadership training our people all the time in the station and of course at the Fire Academy my professionalism was unfounded you just don't get more professional than that org - yeah we worked really hard that being said we did do a real stuff in the fire service so it's not just a little bit of show-and-tell and silliness we're doing firefighter stuff the stuff we want to talk about today are largely the conclusions of other

people people who are actual experts in the field so did a lot of research and did a lot of reading so there's a thousands of hours of after-action reports and research that went in through the fire department responses in the Pentagon and a Washington DC sorry and in New York and in fact the Pentagon the response in Northern Virginia has become a textbook response on what goes well in an incident response these slides will be available later for those that are actually interested in sources and also I have a little bit of relevant training on Incident Response and incident management from the fire service perspective so here's a couple classes I've taken some of them are

actually a week-long and that's a a week-long on incident management it's a week you'll never get back that said in my current role is to see so it's probably the most relevant training I've had so far and most of the training of course was in the National Incident Management System in the US did of a disclaimer the day started with an American it's not ending with one I sound like one but I'm actually Canadian so calling me an American is for those of you or from north of Hadrian's it's like calling you English so you can't blame me for Trump and that's my only political joke for the day so let's start with FDNY a little bit of

background on the incident self September 11th was the largest fire service response in the history of the United States of America but before we go in a little bit of details and a little bit of groundwork I'm sure you've heard the phrase single alarm 5 alarm three alarm can you ball watch TV which shows that have firefight in New York a single arm gets you three engines two ladder trucks and a battalion chief which is the guy little fire SUV who kind of tells people and points at things and gets their picture in the paper pointing every time if you want to get your photo in the paper and a fire scene pointed something while

leaning on somebody they've worked it's magic I've even done it a typical five alarm fire in the New York area is forty four units so for the math nerds in the room of course five times six does not equal forty four but you end up getting additional units in there you get light and air units things that provide more lighting on the scene cylinder changes for your bottles which it's other support units and it's often more importantly a canteen it will typically have stale snicker bars and a watered-down coffee but you're actually I'm always happy to see the canteen is probably your favorite minute on scene the September 11th response included two hundred and fourteen units so there was

a lot of people who arrived on the scene so if you can imagine each engine had four people in it each ladder truck had five people and so forth you've got you know a couple thousand people there running around doing all sorts of stuff so can you imagine running and commanding an incident like that be very complicated and very tricky when we so when I supposed to have a slide here no reminder never mind I remember what I'm doing now so some of the issues with New York I decided to keep it in New York so they had some communication issues and part of the e issues let me just throw back the information where I am all right

I should have reviewed a little bit more in I apologize so these are the main issues that they had here there's six issues and communications I'll talk more about this you guys can read it all of the slides here and these are if you think about how many people have been involved in breaches or cyber incidents in her own world does this look this looked pretty familiar to you I think it does it does to me so common problem with some comments let's talk about communications so for September 11th there was not a protocol in place so when units were dispatched they didn't respond on a radio to say hey engine you know four is responding we're going to

miss so there's no confirmation for the dispatchers to know who even actually received the calls and went to the scene so problem number one of course you don't know who's there but that ties in no accountability via communications and themselves were a problem these are old-fashioned radios they work their analog radios they weren't the 800 and 900 megahertz radios that are in use today with encrypted channels and all sorts of cool stuff with built-in repeaters these these radios were low low range of radios that needed portable repeaters so you have to bring a repeater with you and fortunately a lot of those repeaters were set up in the World Trade Center largely around the command center which was between towers

1 & 2 so we can see some of the issues where that would have come another problem they had is the language they spoke when the fire service in New York they used something that's called 10 codes so instead of saying we have a fire at a single-family home they would say we have a 1017 which is great if you know what the codes are but now if you're trying to talk to the police department the Harbor Patrol the FBI the other units that are on the scene and you're used to communicating in 10 codes that's a bit of a problem it's we don't speak the same language as everybody else there and of course in a security

industry we've never been accused of that ourselves people understand everything we say in our world right so that's one of the other things the other problem we had is because the chief because of communications to people who were running the incident didn't know really what was going on very well they're in their command headquarters the incident command post as I mentioned was between towers 1 & 2 as part of the World Trade Center complex they've even know what was going on very well because they couldn't hear four reports from the people up in there different levels in the building so a bit of a problem how do you steer and manage an incident where you don't know what your teams are

doing and you don't know who's where what's going on what resources you need are you making progress how you're not making progress normally at a small house fire you can tell how things are going about watching the smoke if your smoke turns white that means this fire is being converted to steam or the what and you're making progress if it gets black and thick and becomes more violent you're losing the battle it's easy to tell here they didn't they didn't have a really good way of knowing and then of course when the towers came down that complicated things as well so you know we don't need to talk too much about where we went on

that fact the communications were so bad that even when the first tower collapsed which was tower two folks are in Tower one didn't even know about it and you think they would know right because if a tower collapsing would be credibly loud but fires Believe It or Not are really loud you've got the fire is a loud thing and then of course you shouting each other you're on your breathing apparatus yes you've got adverse conditions these buildings would have been filled full of smoke because of the fire with the airline fuel you know so you've got all those conditions communications and official awareness is also very very limited so there are a lot of issues and

problems with communications now self dispatching is another problem this is self dispatching is when people just show up and there's like well-intended like I have skills I'm a firefighter by gears in a truck I'm just gonna show up and help people did that the other problem that we're going to talk about is recall and that's that that's the process by which you get other people to show up so people who are off-duty it's your right how do you get in how do you notify them to get them to come to work common problems that we have in our incident programs also in a cyber world hub lower self dispatching wasn't too bad an issue in DC it was horrendous in

Northern Virginia we'll talk about that in a second I'm only four units actually self dispatch two of them responded but they get on the radio and said we're going to this put us on and dispatch said no you're not like yeah we are there's a bit of a back-and-forth and so they went a couple of other units people only knew that one of them was there when the engine crew radio say hey run 236 for we've got a few victims we need help getting them out and people like wait who are you where what's going on and that's just not the sort of thing you want my number two thanks for coming so the recall procedure was also a bit

of a problem because New York they hadn't tried it in 30 years so it's just okay don't work nope nobody knew how to do it so being creative and of course it was on the news they asked the media to help out and the media actually put a call saying hey FDNY personnel please report to your normal station your normal duty station and as a result thankfully because it was around shift change a lot of people were able to get back to the stations and they're able to police a a whole lot of apparatus any questions so far feel free to make this interactive if you have questions as we go happy to hear them because I'm sure you want to

hear other voices other than mine now thank you for the courtesy laughs all right moving right along so accountability is basically making sure you know who's there and what people are doing so one of the problems we had in New York is people were told to stage so imagine if you have 200 fire trucks showing up where do you put them that's obviously a bit of a problem because fire trucks are big especially in the States so people were told to stage in certain areas but they didn't they went to different places and that made a problematic tracking who was actually arriving and who was there and when you don't know who's on your scene

or who's available and if people are there and you think people are there but you're not sure then you end up calling for more units because you don't think you've got the right number of people which meant fire houses had less people to respond to the other things that we're going on in City because of course New York doesn't stop even with something like 911 happening there were still fires there are accidents the fire department was still very busy that day although interesting enough people actually called in were very apologetic for calling 911 one that afternoon because they knew they were busy so that's actually kind of nice so yeah as I just mentioned so a lot of units went

to the World Trade Center that wouldn't necessarily have had to go so that depleted resources and other needed area is putting the local population in harm's way and the other problem we had is incident command post as we mentioned it was part of the collapse all of the records for who was on scene all of the fancy command boards where they track all of that was now buried in thousands of tons of rubble so the one source that they had to track everybody nowadays it's easy it's all computer run you have a mobile dispatch turtle on your terminal on your fire truck when you respond and hit a button and boom computers know where you are they

already know your staffing it's all done your accountability is taken care of you're not GPS tracking so it's there's even newer models so you don't even have to put a button cuz like they push the button they say they know you've left the station we're marking you is responding so you don't even have to talk on the radio anymore much easier back then you had to talk on the radio and of course was only six single-channel and it was very complicated so imagine if that now situation you've got two towers that have gone down and you don't even know who's there bit of a problem situation for New York and so when people were managing their

things it was very chaotic and ad-hoc so which is basically oh you outrank me and what do you want me to do there was no coherent plan and how they are moving forward it's the day started with one but due to events in the unforeseen collapse of the towers nobody knew what was going on or really how to control or manage an incident again something we've kind of seen in our industry as well so the chain of command is the other part is who do you actually report to so the first question in a breach at your organization's do you know who is actually in charge at my back you know I look at their management team look to me

after see so they say ok you're in charge and I respond no I shouldn't be but do I have the authority at the bank to shut down our trading floor if we have you know say a significant situation where we could say we might exfiltrate two to three hundred million euros of money that day because of nefarious things that are going on sure I can shut down our trading floors what I have a job the next day probably not you know so those are very senior business decisions that I have no base is doing my job is ultimately to steer and manage an incident but we'll talk about that again in a second so people

had conflicting instructions as you're gonna get in these complicated things there's a lot of chaos going on between the noise the collapses the broken communications it was if you're part of the expression it was a show there was not top you know loss of life's not awesome life notwithstanding it was not a good day people were told you had multiple units doing the same thing you had things that would need to get done that weren't being assigned which of course now puts more risk to the firefighters because you're putting more firefighters in harm's way and you're not getting the firefighter and life-saving resources that's where they're all made it to help do civilian rescues so again putting people in a bad

thing here we have largely talked about what we have going on so we can skip that slide mutual aid mutual aid is basically the engagement of other departments now FDNY is one of the largest are I think it is the largest fire department in the United States the thought that FDNY would need to call on surrounding jurisdictions for bodies and tools and fire trucks never really occurred to them so they didn't have you know they it never it never happened so they didn't have a process to even determine what situations need to happen where we need mutual aid how do we need other you know who are we going to call how are we going to do it

they didn't even have a means to call people they don't know who to call and what's wow there's a type of interesting and once they were called how do you integrate the people into the local operations you know these guys they don't know your language they don't know your ten codes they have different staffing what could be a heavy rescue in one jurisdiction might not be a heavy rescue in another people have different definitions for things so these are some of the common problems that that happened there's and when you repeat when people calling requests it was just it was unclear because of these these problems and the other thing they had is they didn't even know what yet

surrounding jurisdictions could offer so also problematic hey we need more tankers because you know we don't have an underground water supply anymore where do we get them these were problems that folks have to solve on the ground so not a good day for the fire department in New York for a lot of things but let's try moving some little lighter let's talk about where things started going a little bit better in Arlington County Virginia which is home to the Pentagon Arlington is a bit of background is the smallest county in the United States so I like 26 square miles or for those of you that like metric 67s where come whatever opinion is their entire department consisted at the time

of 292 firefighters so if we look at the New York response literally ten almost ten times as many people responded in New York as they were on duty or could be on duty in Arlington they only had ten stations that are staffed 24/7 with 13 paramedics and in 2001 the council of governance I apologize for the acronyms but just keeping I believe in left text and more pictures the council of governance which is basically the group that managed the local counties they decided to adopt some in the NIMS ICS model which is the National Incident Management System incident command structure so remember I show that long list of all the crap I've been trained in that's what this is in

fact after 9/11 it became a federal requirement for any department than one in federal funding but all of their personnel had to be trained in a standard incident management framework which in this case is ICS so northern genia about six months before they decided to adopt this standardized framework on how to respond and it wasn't just in Arlington it was the neighboring counties was Alexandria and was Loudoun County where I was a fire fire firefighter with Fairfax County Stafford County in Prince William County so we're known otherwise known as the Nova cog so if I talk about Nova region by being those five counties and I am going to talk about Nova Region so you

know I buy a beer for anybody who remembers the name of the counties so this was very relevant because at the 911 response at the Pentagon and challenging I can't give you exact numbers here and we'll explain why in a second there were thousands of people there so it was a big deal happily most of those people who showed up had a standardized way of working so some of the things that went well and went bad we had a lot of these you'll see as a common theme and again common to our own world in our old world in cybersecurity so self dispatching this was a big problem at the Pentagon in the Northern Virginia region there were tons of all

into your fire fighters myself being one of them who decided to throw their gear and their and their pack of their pickup trucks and add to the Pentagon because they want it to help of course when it get there they've got their gear but they don't have breathing apparatus they don't have any tools and nobody knows who they are what their capabilities what they can do what they can't do what they're trained in what they're good at what they're bad at there's just somebody with a fire man's hat and they showed up to help and at that point into the first couple days like sure come on in help because you they needed it it got a little more

structured which also became a bit of a problem later the incident commander James shorts he ended up installing a fence around the Pentagon to stop this from happening because this is also the Pentagon this is the headquarters for the United States military you need as a general rule the minimum was secret level clearance just to get at the facility to walk in the door actually it's a working sub-basement in north wing well that's a story for another time so they had they also had a big problem how do you track who was there when I said at the beginning of thousands of people showed up still today they don't know who is actually at this response a

bit problematic because they're handing out really cool ribbons and awards and stuff like for hey thanks for coming people got letters from George Bush at the time which they actually appreciative of those letters has no political joke intended and so that was great but not everybody who showed up was able to be thanked not everybody who showed up was recognized so it's still problematic because there are going to be medical complications as a result of the things that have happened the byproducts of these accidents to buy part of the incidence of my products have been complete combustion are highly carcinogenic and they also do really bad things to your lungs and all sorts of different ways there's all sorts of

nasty stuff in there and firefighters more firefighters have died since September 11th because of September 11th but that's again that's a that's a story for the pub not for here I'm you know I'm trying to end on a real positive note kind of not happening right don't worry it gets better so and the other problem of these additional security measures if they actually impacted who was getting there they ended up instituting badge checks and of course the FBI at that point and he's gone out they you know taking a little more lead in what was going on they started saying well we should change the badge colors every day which is problematic when you have people

coming in and out literally by the hundreds in some cases thousands so shift people would finish those shifts we were waiting three or four hours sometimes for their relief crew you come get them so people were working extra time and pretty hostile conditions putting themselves to create a risk of cardiac episodes and things like that which are actually the number one killer of firefighters in the u.s. random facts on average about 100 firefighters are killed in line of duty in the US every year the numbers been pretty consistent more than half of them are for cardiac related incidents it's not actually from the structural collapse or fire related things it's largely because they're out of shape in

their heart says yeah I'm done but if you think about it you go from sound asleep so three minutes later you're going boom down the road full when you're geared to go do your whatever it is you got dispatch to do something pretty stressful on the body I mean heck I'm only 24 sorry anyway so going back to command and control we had a bomb prevented and interestings a situation prevent presidential decision directive number 39 says the FBI has jurisdiction on all terrorism attacks on US soil of course the attack took place I see what went on my pictures over the text the text the attack took place at a Department of Defense facility being the

Pentagon of course the headquarters as I mentioned for the US military so lots of interesting information there and a big hole in the building and of course Pentagon security forces it's a separate group the works for Department of Defense their job is to secure the perimeter of the Pentagon make sure people go in and out as we mentioned the FBI had jurisdiction so it problematic however everybody played nice together so because of the integration of the the incident command framework can we talk about ICS which we'll talk a little bit more about that towards the end the the folks there actually had good relationships they knew each other people knew who to talk to and it went

well so that was a nice thing the fire service stayed in charge while it was the fire suppression activity and then the FBI took over when it became a crime scene investigation and the chain of the command was smooth and very well-defined and had actually largely been agreed on before this happened communications here was also a bit of an interesting problem by the way for those of you that are history buffs and I doubt there's anybody here in this room know what this is I wouldn't expect you to but I thought it's worth asking this is what's called the minitor this is what you would wear if you're a volunteer firefighter or an off-duty firefighter

it's kind of like a pager it would make it really annoying beep but only for certain units and for certain stage station so it was actually mapped to your individual firehouse so if your fire station was dispatched on something is back back in those days not all the stations had 24 hours staffing a lot of times the firefighters at least in the volunteer districts would sleep at home because they weren't running calls every night they weren't super busy so they carried something like this it would buzz and then they would hear the dispatcher radio announcement they would go to the firehouse and they would know why what piece of equipment anything they'd know where to go really cool a

little bit of fire history there that actually I've broken several of those firefighters played Kings so part of the radio issues of course for the Pentagon it's a secure facility radio waves are by design not supposed to be going places so you really have to rely on basically line-of-sight communications on your radios and even then it didn't work very well so that was a fairly problematic so they decided well let's use cell phones of course I had a big incident like that the cell phone towers get saturated since 9/11 they actually have priority numbers and every member of a critical infrastructure team has a little card I actually had one of my wallet with a special number to dial and

all that kind of stuff that's a priority number so you can actually use a mobile communication system when they're saturated if there's a large scale event of some kind going on so but cell phones were working pagers actually worked really well but not everybody had pagers so bit problematic here again the communications as a recurring theme not a good thing the other thing that went wrong at the Pentagon was the lack of a joint information center as a rule of thumb when you're doing a joint command with inter agencies you end up having an information center whose job it is to talk to the media because the Pentagon would normally would have normally been staffed with several thousand people on

the west side and you know they're expecting several thousand casualties and that was what was being reported in the media it was less than 200 fortunately but so there was a lot of misinformation going up there which was of course causing additional panic and drama with civilians and that's just not a good thing you always want to control your message and yeah talked about that and the other probably of course with it we have the and on one of the normal things you do in the fire service is you get to know your response area you know for example if you've got a shopping center you know what all the entrance points are you know what to call them you know where to

plug your fire truck in the get water or to push water you know where to get it you know where the connector you do what's called pre-planning of course it's a pentagon they're not really inviting the fire department around so don't do all that kind of stuff they do now but they didn't then so this was also a bit of a problem for obvious reasons questions where we are so far all right still with us you wanna go to the pub I do see had no it wasn't so the question for those who didn't hear it apologize for the oversimplification was ultimately who actually ran the incident because over here it's a little different and how it played out for the

incident commander for the for the first few days while they were doing fire suppression was a Arlington County Fire Department assistant chief Schwartz he was the formal incident commander for the large majority of the incident there was an FBI agent whose name I don't remember who was actually working side-by-side with him as a primary liaison but overall the fire service while there was still fire to be put out and a rescue activities to perform the fire department was in charge once the the fire risk was taking care of the risk of structural collapse in fact they called the evacuation tones four times which is frustrating for the firefighter you make progress putting the fire out

then you have to evacuate and then the fire goes back where you were and allow me just work for four or five hours for nothing you go back and forth but uh fire the FBI took over what's once the fire element whereas without anything else and while I'm going I need to speed up sorry so the contamination is a bit of an issue at first because you've got all the bad stuff in the air and when you're dealing with a lot of unburned jet fuel which is also toxic you have to be contaminant your firefighters there when you have a lot of firefighters there there's a lot of infrastructure needed in an event I've got a really cool picture towards

the end that actually shows the setup the use of the Pentagon for the infrastructure batteries you don't think much about batteries you typically have a box light on your fire truck for each firefighter that that light lasts about six hours which is great for even advanced operations now the Pentagon of course you know there's no lighting if the power's out for obvious reasons you know they needed batteries in fact they call the local Home Depot which is the equivalent of what's the hardware Center here again that's Orange or so yeah thank you that one and they call the hope in about 10 10 minutes at 10:00 p.m. you have actually open it useful times over there which is helpful I live

in Belgium stuff closes at 5:00 and 6:00 it's frustrating but so they call that about an hour later a truck with every battery in the shop showed up so kind of cool to see the community come so but logistics was a is a huge problem in a large scale events air supply is also a bit of a problem and I don't mean that great eighties band I put this up here to make sure you're all awake you know I'm all out of love and so are my bad jokes but moving back along air supply was an issue because when you have that many firefighters during fire suppression activities and you're on your apparatus apparatus you typically

get depending on your fitness levels and activity levels 30 to 60-minute use and when you're talking several hundred people doing operations on the ground at any given time you need quite a bit of infrastructure to refill those cylinders to complicate matters not everybody in the Northern Virginia District use the same brand of cylinders so some people had a company called MSA and other companies use a dragger which is German company a lot of folks you Scott and so that added to the complications again I'm still going back to the air supply the badging we talked about how do you actually identify the people that are coming in and out that was also problematic there nowadays is simple you get a badge it

has a little two-dimensional barcode with your nationally accredited records and certifications on there so you can show up at a scene they can scan it with a certain kind of reader and they know you're trained in hazardous materials you're trained in technical rescue you can drive fire trucks aerials well all these things or you can't do any of these things now there's a standardized approach in Northern Virginia that's actually all in everybody's ID card so that's actually really helpful especially when you're doing mutual aid and and complex situations on a little fun story this is a story one of the problems with the Pentagon is one of the main access routes had a stone foot

bridge and that foot bridge was low and to do there's a in the fire service if you've got big flyer you want big water so that's where you use typically a tiller truck or a ladder truck I do you guys know what tiller trucks are I didn't think so so here's a picture of one this is the one it's the classic fire truck you see with the driver in the back we're good it does so pretty tall you put out somebody who steers at the back and you know the big rig at the front so these guys it's a small volunteer fire company this little fire truck picture here it's a 1955 b85 Mac tiller truck unless you're a nerd you

won't know anything about that but it's really cool it's an old vintage piece but the cool thing about it is called an open cab fire truck they're not allowed anymore because of health and safety regulations the bottom line is all that top stuff wasn't there and it's because it's older and fire trucks have gotten in the u.s. much bigger this fire truck somebody knew about it and they called for and these guys got dispatch drove all the way to the Pentagon going about 65 miles an hour because that's as fast as the truck would go downhill with a good wind and it went under the bridge and they were able to flow about eight nine hundred gallons of water per minute

so multiply that by four roughly four liters per minute and so that helps until Arlington decided to cut the cab tops off of there tiller truck and then brought it in because that one could be about 2,000 gallons of water per minute so they got very creative and how they solve some of their logistics of rotary k12 saw a little bit of willpower by the way those fire trucks costs almost a million dollars I would love to be the guy who saw I gotta admit so anytime you're doing extended operations support you need toilet facilities the canteen I mentioned all of these things are problematic and also relevant and large investigations and cyber incidents at our own organizations

you can't always deliver pizza deliver to two o'clock in the morning and Haken that part of the world I live in nothing's open at two o'clock in the morning it's not even open outside I mean not even churches were open on Sunday in Belgium it's a sorry Pat joke again mutual aid as I mentioned we have the COG agreement in place so all the fire departments we had standardized playbooks which we'll come to in a second there was already an exercise planned for the Pentagon so there was a lot of homework already done on how to respond to a big event at the Pentagon so Arlington County Alexandria County knew how to make stuff work they have

standardized response playbook so I'll show you an example of that in just a second also have standard communication protocols each County is assigned with a number of my fire my County was number six so our fire trucks add six aza fire station 11 it said six eleven on our fire truck our radio channel was six really you got to keep it simple for firefighters and it works so we all know how to talk to each other we knew if we talked you know if engine 514 called we knew they're from Prince William County it's a really easy way of knowing who they are and we had standard builds in terms of what all of our units work also

back to this one of the other things that was really cool is all the supporting jurisdictions in the area basically called Arlington hey whatever you need just say the word we'll have it on standby thing I love about the fire service is it it is truly a family that help each other and it's it's one of the things I miss so moving on to things let's lighten it up a little bit and let's talk on how to have our own incident responses and our own world's take some of the lessons from September 11th and actually get some benefit from the things that went horribly wrong but also the things that went well so the first one we need to look at of

course is command and control we're going to talk about communications and training and the exercises associated with this all of these after-action reports identified about half a dozen elements and they were all common themes but I think the three most important ones are these three so command and control in order Virginia we have a manual that has the talked about how to do command and control how many of us have where we work have infinite response plans and stop please help me out here Wake optical alright how many of you guys don't have incident response plans where you work okay that's should've been the better question all right because that means you don't have to put you at most of you

don't put your hands up alright so I'm happy with that do your incident command modules have a detailed basically process and procedure on how to manage an incident so we do and I'll show you what it looks like but the first problem is who's in charge as I mentioned it shouldn't be the see so in my opinion we shouldn't be in charge I'll talk about where we should be in just a second at my old bank where I was working in the u.s. our general counsel for significant events they were in charge because at some point you want the lawyers making the big decisions span of control you don't want more than five to seven people reporting to you at any

given time on in a crisis situation because it's hard to manage resources more than that there have been lots of workplace studies five to seven is ideal for those of you that have more than that report reporting to you at work you have my sympathies and I've got the same problem chain of command is also a bit of an issue and that's really whose instructions do you actually follow if there's a cyber breach at your organization who is actually the person you should be listening to if you're running the saw or if you're running the team they're doing their forensics who is the one person that should be telling you what you need to do next is that

clearly defined these are the three these are things that you really need to address so the other considerations with that you want to have your crew on a general crisis management framework in your organization the u.s. is National Incident Management System is I think a very good crisis management framework here in the UK and across Europe we do gold silver bronze three-tier things which are great for what levels but they don't actually go to their granularity on an incidence actually how to tactically manage things in a broad way they're more operations geared and definitely more for law enforcement it's not a good generic framework this is just my own opinion here those of you who've been involved in those things or

use them I have made no offence but a generation Eric framework is important because you can activate the pieces you need and you don't and don't activate the pieces you don't so it needs to be very modular and more importantly you need to have a list of well-defined roles you need to plan ahead what are all the functions that we are going to need in a worse case of event how do we define those roles how do we train how do we identify who's going to be in those roles and then how do we train those people so that when an event happens they actually know what to do and I highly recommend that you separate

the management sort of y'all from the operations side because the last thing your forensics teams need is the CIO the C so the CEO somebody's showing up hey how long it's going to take because on TV it takes five minutes you know some of our queries of course take the days to run like if we go away if we have a situation at my bank it can sometimes take us days to run queries to find out what's the potential impact of certain situations it's been a tough battle to get management to understand that that's how the real world works CSI screw you you have really ruin it for all of us it just doesn't help so

this is this is the National Incident Management generic command structure this is how they manage all incidents in the fire service this is what we use the FBI this is what they use as a bit of history this command structure was actually developed by California firefighters as a way to manage large area fire responses and it's now the standard for the u.s. you really got four main branches just at the top you have your incident commander generic term but that is the person who's in charge that doesn't have to be the most senior person but it has to be a person who's good at coordinating and organizing and managing things you've got then your public information officer in our world

this would be a communications department these are the people that'll talk with your social media these are people that yeah basically do your outbound communications you have somebody who does liaison in so if you are getting like calling a man from police Scotland to come down and help you out you know we want somebody there to interface with police Scotland or if you've got third party companies or four vendors involved you need to have a dedicated person or a dedicated team whose sole job is interacting with the other organizations you have there we have course have a safety officer in the fire service because we're firefighters and we we do silly things and yeah I'm I

have multiple injuries I kept showing you those pictures from stuff I've done on duty that's really stupid Beth one getting carbon monoxide poisoning wearing a Santa suit on top of the fire truck during a Santa Claus Parade that's why we have safety officers people like me but moving right along to the core part this is split into four sessions you have operations these are basically the people that are putting water on the fire the ones that are doing the real work so this would be in my mind your sock manager or whoever's running your security operations on the planning side of things that's I think where the CSO should go this is really what are you

doing over the next 12 to 24 hours what are your next steps how what is it you need to do logistics is straightforward once you decided what you need to do their job is to figure out you know how to get you the supplies if you need more people if you need more tools if you need Pizza whatever it is the job of the logistics branch is to solve that now finance of course are the unsung heroes let's face it if you have a significant incident the purse strings open up and checkbooks are there spend what you need but an order you better be able to defend every pence you spent on the back end so get the finance

people involved right away so you can have that paper trail procurement we'll make exceptions if you don't have an arrangement with a third party and help you with your breach responses do it now because it'll be a lot cheaper if you have a contract signed and the day off you know there's a difference between a digit and those daily rates typically so at a high level this is very generic command structure does this make sense for you guys pretty clear I like this we've actually mapped this and at my back my old back we are currently mapping this right now where I work to do our incident management and incident response so command and control

they need to be well defined but they still need to be generic you have to train people that I wanted to end on a lighter note so the pictures are gonna be more fun this was actually a joke picture by it was I think in Norwegian Fire Department these train tracks weren't in use but they're not training I thought that would be funny it was of course a viral photo but I just loved it you have to train people make sure that the rolls that you do are also mapped to how your organization works you want to be you want to have some kind of alignment we relied your contact information make sure that it's verified there's nothing

worse than trying to call people and they don't even work at your institution anymore not helpful so have multiple alternates offer all of your roles and of course with a with these structures you only activate the bits you need if it's not a big incident don't activate everybody communications very straightforward first of all manage with your expectation or expectations with your executive teams talk to them about how long investigations take if it's not TV that will save you a whole lot of time so that they don't fall in your ever behalf now are looking for updates when you're trying to do your job you know make sure internally you have a formal way of communicating with your executive

team and that they know what it is and that they bought off on it for your external communications already go through legal and your social media departments and all that have stuff that's pre-approved and have fill-in-the-blank information so you just you just put in some values and nobody's arguing about verbiage and today's social media world you need to respond quickly and you need to respond accurately remember the Tesco breach from about a year and a half ago it hit the news on a Saturday was gone on a Tuesday they managed it well then of course there's talk talk exactly so we taught social media expectations we've talked about that but make sure you've got people who are going to

communicate internally and externally and if the corresponding and partners know who they are and also within your own institutions have a redundant communication means because it could be your network is down and if you've got voice over IP phones that's problematic if the bad guys are in your network and you're managing your call out with a VoIP phone that could also be problematic depending on how skilled they are so have good we're done and out of means out of and means of communication and also have multiple channels for the different functions so you're not over saturating in one communications mean training it's straightforward as I mentioned try to separate operations from the executive piece because there's a difference

between you know standing at the fire SUV and pointing at things and saying put water there and actually knowing how to pull your hose off the fire truck and how to operate that effectively do tabletop exercises I highly recommend you do them at least quarterly because it verifies that your processes work it also gives confidence to your management team that they know you've got the ball and more importantly they know what they need to do as well and it also is a good opportunity to get more funding for shiny things typically and we all like our shiny things when you do your training don't insist it everybody is there because not everybody is going to

be there when something really happens so it's okay if key decision-makers aren't there make sure you've got some alternates because that's how it's going to play out in the real world and also track the lessons learned and have that feedback loop for those improvements also where possible if you're a large organization coordinate with HR so when you're hiring new senior people you get an opportunity to sit down with them if they've got a role in and it's at a response to the organisation that they know what that role is and try to do that right away they'll say it's a great opportunity to meet them gain some political credibility and more importantly it could be helpful when the

bad stuff happens and on your day to day activities try to use the same structures as you would in the fire department we had if we had three units so if we had like an inside gas leak which gave you two engines in a ladder truck we use the incident command process and we invoke that structure so that it became second nature to us we use the same terminology the same language and it was a very effective way of doing stuff and our sock in our incident management teams our blue teams are now doing that in our own bank and we're doing that now to help them get used to the language that's been very

helpful so the important thing train the way you play because there's you know different ways of putting out your fires how much is too much and I know right now we're you're looking at your clocks in the fire department in Northern Virginia these are all of the manuals we have these are all the different play books for the different kinds of responses so as a firefighter you need to learn your job as a driver you need to learn your jobs and all of these as an officer you need to learn everybody's job a lot of stuff to learn and these play books are detailed with your playbooks they need to be easy to read you need to be able to find them don't

store everything online for obvious reasons because you want to be able to be able to get at them if stuff goes horribly wrong in your network and also they assume that people know how to pull the hose off the fire truck so you're not saying this is how you work dark Trace you don't want that level of granularity but you want to people to know do the following things and again with your table tops review them or regularly you can never train too much so let's just very quickly on the command structure this is the index of the incident command manual so we've got just under 170 pages on how to manage an incident and it talks about all the

roles that are in the incidents so it's very thorough and very defined so if you have a playbook on how to manage an incident and it's half a dozen pages I promise you it's not enough it's not well thought-out it would be well intended but it won't be very effective so that's what it looks like on how to manage an incident for operations this is a sample from one of ours in terms of it's like about 7080 pages and that's for a specific type of fire which are the mid rise building fires which are buildings between 6 and 10 stories so there's a lot of information and that's how to put out a fire in that kind of

building so let's recap what are the key things we've learned on command control we want to define who's in charge understand your chain of command make sure you have well-defined roles and don't have a span of control that's too crazy you don't want too many people reporting to you communications use the same language use normal English wherever possible avoid geek speed don't use lead speak nobody wants that anyway that's that's like 1990s stop it but use normal English language because you'll be interfacing with third party a or agencies and vendors and they might not know your acronyms they're not going to sort of try to be very common and practice everything you do as often as

you can because in the fire department you don't just practice pulling your hose off once if you're not busy that day you're pulling hose you're throwing ladders and in our world we need to be doing the same thing every day so that is the picture that I had promised of this with the logistics setup in the parking lot at the Pentagon a lot of stuff going on it has showers canteens dining halls lots of toilets all sorts of things because they're literally couple thousand people there every day so that is how to run a large complicated incident which i think is really cool are there any questions as a point of information just before that

fire was taken I was whoops here there's a great day we got 11 save one goldfish into cats I care more about the cats I'm sure the goldfish was happy people you can always make more babies but um anyway questions sorry we're totally running late yes that is always the tricky part the best thing you could do is if you can try to find an executive sponsor and when somebody is a senior person on the business side in your organization and get them and just target on them because in most organizations generally somebody who kind of gets it get them to send out the invites the most thing of the person that sends out the invites that's why

you get your people if you're just coming out from from a god I'm a fairly senior person in my own organization if I send them oh people are going to show up but if our CEO sense about people show up so you have to be very strategic in building those key relationships try to find a stakeholder that doesn't have the word security in a title in fact don't ever use that word it's a bad word stop it it doesn't it doesn't help talk about value talk about risk stop saying security but that's the only generic advice I can give we can talk after I get a little granular information on your organization and now people little bit more that way yes sir

right so to make sure I could hear that I think the question was should we have playbooks for things like DDoS is malware yeah yeah short answer is absolutely with all of these things there are standard repeatable tasks that you'll do to manage a DDoS attack or a malware infection or a phishing incident or some other indicator of compromise there are things that are pretty standard so those you should document and make repeatable and make sure you know who's doing what because part of the problem is okay well we've got this okay now what do we do so it's great to open up a little binder flip through pages and let's do this unless the

situation warrants something different and the playbooks I showed as the first arriving officer you have the right to change what's going on because you're the you've got eyes on scene and you'd say okay we're not doing a normal player we're gonna do a B and C you get on the radio and you communicate it you do the same thing in our own in our own world as well I see you in the corner of my eye Rory but I'm still asking for more questions anyone else okay we had one last one right the simplest way to answer that so the question is if you're a smaller organization who's going to do the night work who's you know you don't

have all those resources have read people 24/7 it's just summarized and then for the sake of the recording as well is that a fair summary of your question okay so the simple answer you're going to need external parties you there's no way you can do it yourself there's no harm and basically putting a standby agreement with a third party firm so if we need you you know this is what it's going to cost this is what we're gonna do your I do recommend if you do use third parties you have them come when you do your training exercises so they're familiar with your organization a little bit as well you of course have to pay their consulting rates for that

but you know really you can't do it yourself if you're a small team and here's your point yeah per note it's it's brutal and you're getting that you're gonna your teams are gonna be there anyway because that's just how people are so make sure you have some food on hand and don't do it in Belgium I thought bias against Belgium it's a nice place Rory yes you have things to say yep we are sorry so first off and I thought I was going to be done quick