BSidesCharm 2023 - The Action Group Model for Incident Response - Shawn Thomas and Taylor Johnson

foreign

[Music]

all right so I got kept up till four o'clock in the morning last night I feel like I have more energy than all of you so can we try again what's up besides charm that's much better how are you all doing today my name is Sean uh it is bring your boss to work day so I brought my friend Taylor here hey how you all doing Taylor would you like to introduce yourself sure I'll go ahead and introduce myself so uh I have the wonderful wonderful amazing job of being Sean's boss it's exactly he's lying I don't even know how I have energy this morning to be honest taking care of him but uh no and in all honesty

Sean is amazing to work with uh we work together uh at uh at Yahoo we're with the paranoids uh before that I did about 10 years with the FBI doing some counterintelligence type work it was super secret and super fun um but uh yeah I had a good time but here uh to talk to y'all today and uh go over the Action Group method Sean yeah he also left out so he is the director of threat detection and response at Yahoo uh I am the director of forensics and incident response engineering lovingly called fire internally um before that IR consultant sock work a whole bunch of stuff so we have spent the last couple years working on what we

call the Action Group model the incident response we think it's kind of a cool thing so we figured hey we're going to come talk to a bunch of people about it because it'll be fun yeah it seems to work okay for us so hopefully you know this might be interesting for you all maybe maybe not we'll see take what you can home with you yep all right you wanna you wanna Dive Right In yeah let's do it John cool all right so we're going to start you off so this is our general workflow for the action group model so essentially this is not terribly different from most other models of instant response you have sort of an

intake section hey how are we actually bringing things into uh our our area um are we finding detections do we have things that are being sent in by our partners we have to do something with that after we have it though this goes into our triage section where we're really asking the basic facts of what's going on now our folks that are in our fire area essentially doing these first two areas they're receiving things in they're then asking the basic facts and trying to determine do we have a problem here fundamentally that's really what we're wondering and really what we're looking for is is there some sort of malicious activity um that could be going on and do we have

something that's a big problem that's what we mean by imminent material compromise that we have up there on the board right now but if we do have those two things we go into activating our Action Group if we don't have those things this is our normal BAU stuff hey looking hey we've got maybe a fish email or something like that we need to deal with doesn't look like there's anything going on whatever we can handle that normally but when we do have something that looks like it could be a big problem we move into the action group and this is really the gist of what we want to talk to you all today this is

where we bring in a number of different folks a number of different partners from all across cyber security Human Resources legal ethics and compliance physical security to work on these big major incidents when we do run into them and this is really important we'll get into the details of these in a few slides but as we go through this we kind of continue on through the general method we do our response this is where we're asking what are the key questions that we need to deal with here uh and then we then go through set for those work streams that we have afterwards we go into our closeout this is our normal Lessons Learned post-incident response activities and

things like that but as we go forward and as as Sean starts to walk you through all these I want you to be really careful and be thinking about sort of this Action Group model bring a number of folks together having these pre-established connections to be able to come in and provide oversight and assign things to a dedicated Incident Commander that can be really any of the individuals that you're working with so anybody that's working in a sock in our area anybody that's working on our fire team can step up and be an Incident Commander with that guidance of that action group that we have across the organization you get over you Sean yeah go ahead jump

to the next one so we'll go through each one a little bit and something to bear in mind your mileage will vary we actually tried to extrapolate our plan out into a much more simplistic concept that can be applicable to your organization so some of our language is removed and we kind of open this up a little bit so like intake intake is just how you find stuff how does stuff come in do people tell you about bad stuff do you have detections do you have threat hunts do you have a threat intelligence team it's ultimately about all of the different processes and feeds that feed into you finding bad stuff or I should say at this point investigatable things

it's really important to know your intakes because those are going to tell you what work you have to do those are going to tell you what gaps you might have as you look at your intake so on and so forth we're not going to spend a whole lot of time here we'll go right to the next one so triage we simplified this one a little bit too basically this is your general sock which I kind of hate the term sock these days IR whatever case you want to make your investigative teams are going to do this and they're going to start here so first thing they have to do is kind of ask themselves a very simple question

right is this bad that's really the first question if it's not well you're done hey move on to the next thing or you know tune it out there's a whole bunch of other things you can do if it's not bad but for the sake of the IR process if it's not you're good if it is then the next question you have to ask yourself and this is really important is does this meet the definition of an incident at my company your mileage may vary on what an incident is some places every vulnerability that's critical is an incident in some places it's not so your mileage could completely vary but you have to ask what is the definition of an

incident and this yeah it was just going to add in their Sean too this is something you need to be asking yourself before you actually get into something that might be an instant uh we talk about this a lot all the time is this is something that you don't want to be discussing right when you see something bad yeah so it's going to be really important to jump in there talk with your folks talk with your leadership talk with your partners that you have and make sure that you understand exactly what that means for your organization there are lots of definitions of incident out there but as Sean was saying this will highly depend on what your business is doing what is

valuable what your crown jewels are what your critical assets are it's really important to make sure you define that early so quick note on IR burnout a lot of ir teams burn out because they don't have a clearly defined definition of an incident so everything gets lumped on their shoulders and becomes an incident that's not where you want to be you need to Define that very early so that you can say hey this is when we have an incident process because incident processes are time consuming and involve lawyers which is exhausting looking at you our lawyer is here so I got to make some jokes um so if it's not then what we do is we

move it into a process where an analyst can handle it so to use Taylor's early example of a phishing email if it's one email low impact nothing bad happened right an analyst can handle that without a whole bunch of people we don't need to spin up and call in the the strike force or whatever words people use these days all we need to do is just go mitigate that one little issue it's what we call an event rather than an incident now if it's an incident that's where we come into the Action Group concept and that's really important so as you'll see as we work through the workflow you initiate the action group meeting you set up a

comms Channel and you assign an incident commander and we're going to take some time and dig through those components so go ahead Mr Johnson cool so let's talk a little bit about the action group so we talk about this action we've set it up but like what what actually is this and and really what we're what we're looking at here and what we're talking about when we mean Action Group is getting stakeholders in the right place to be able to make operational decisions uh and to be able to provide the right guidance to the folks that are actually working the incident um so this is really really important to establish this upfront and determine who

actually needs to be in the room and this shouldn't be folks that are up in the c-suite these should not be Executives who are going to say okay cyber security stuff bad guy doing something all right uh call the police like not not that right we need folks who are actually at the decision-making level they can make tangible decisions on what's going on but also are close enough to the operational level to understand the ins and outs and the different work streams that you're going to have when you're working one of these incidents and so typically this action group is going to be making decisions on where you're going but really establishing work streams um making sure that these work streams

are are answering the core investigative goals that you actually have and so you know when we talk about who's actually a member of this action group like clearly you need the folks that are doing incident response so um for ourselves you know Sean myself on this um but also you want to have other elements of cyber security that are important to have in there as well such as your intelligence teams that you have uh potentially even a red team point of contact maybe some of your policy folks this can vary slightly depending on what you need to deal with in your organization but having the right folks in that room that you can really get

answers to most any questions or at least know the right direction to go to it's really important so one thing that's actually really important about how you structure an action group if you're going to do it it's split up into what we call responsible parties and support parties and that that is kind of the key because a lot of people when they think about an incident they think about just the responsible parties right the IR team is a responsible party up to your CSO is arguably a responsible party but they don't think about support parties your detection engineering team great support party your threat intelligent function great support legal sometimes great [Laughter] so it you really want to look at like

who can help with this in some way shape or form because as anybody who's worked an incident knows they're all complex they're all different especially like big ones so bringing together a multitude of experience thought processes and skill sets is going to get you through that better and I'll dive into that on like the importance of like variable work streams later but good yeah absolutely and I think if you think about this we take a practical example say you've got a corporate account that's been taken over by an adversary or something like that as you dive into that yes you're going to need to deal with sort of the Tactical of okay how do we evict this actor how do we actually

figure out where this actor is but oftentimes you're also going to be wondering okay what data has been compromised could things be out the door do we have any regulatory obligations and having those right folks in the room that really understand how the process works to be able to give you that counsel really in the moment is incredibly important because one of the things you really don't want to be doing is when you're dealing with an incident go who do we talk to who do we actually go to to get an answer to this question are we under some regulatory obligation how do we actually get this data all of these questions may come up during an

incident and I think it's really important to be able to have these relationships pre-established and make sure that they're all in that room now one of the things that you all may be asking and are probably wondering here is hey we've got a bunch of cooks in the kitchen now so we may have these responsible parties or maybe these support parties that we have in our Action Group too but how do we actually make decisions how do we make sure that operational control is maintained and and for that I'll kick you over to Sean in The Incident Commander concept cool so this this probably isn't a super New Concept everybody but this is something that's super important to talk about

across your organization to make sure that it exists right the first thing that the action group does before we do anything else is we establish this role of The Incident Commander one of the great things about the Action Group model for us is it makes The Incident Commander a very accessible role so it's not just who's the most experienced guy or who is this because they have a team of people that are there to support so we can make it a junior and we often will on different cases and can it makes it a very accessible model because they have support it's kind of like being thrown in the ocean but having somebody next to

you with a life vest that's really kind of my whole model of leadership so seems to work all right yeah I don't know what is the front row say over here yes okay we got one no I okay I can do that um so the other thing that you really gain from The Incident Commander model though is you have somebody who has operational control that's really important and that has to be agreed upon at a business level so when that person regardless of their rank or title hits up a VP somewhere and says I need you to do this they understand that they need to do that that is like the most critical thing during an incident it's also the person

who frankly keeps track of things and helps us sign out the work streams that we'll talk about in a minute and all those other things because somebody's got to keep track keep everybody on task and make sure that we have the power to handle the thing that we need to handle in that moment yeah and I I you know it sounds scary in the moment particularly for junior employees who are going to be an Incident Commander oh my gosh we've got an adversary that's doing something we got to deal with them and I'm the commander looking at you mark over there uh but uh but I think fundamentally that's why the action group is really

important because you do have the leaders in the room who are tasking out those work streams and understanding what we need to do and really empowering that Incident Commander to get done what we have to get done for the incident itself um also really important to bring in say legal counsel with that Action Group to say like hey this is what we can do this is what we can't do those sorts of things as well just wanted to shout out bunny Smith Who's down there supporting us and gave us these these lovely shirts today they give us these lovely shirts all right so this all kind of bleeds into the what happens when we actually

do response though right so the way we do response and the way that we think about response is this very secular thing it's also meant to be particularly adaptable this isn't some Playbook that says always go do this always go do that always go do this other thing I mentioned earlier incidents are different different things happen different things are affected there's different ways that you need to approach it so we put a whole bunch of smart people in a room as the tldr and we had an Ann Taylor and John yeah plus Taylor and John true and then we go what do we need to do okay like we need to know this we need

to know this we need to know this we need to know this we should investigate here mitigate here do this this and this we build that out and then we assign people to it and we assign across the entire scope of the action group so we'll go to if our threat intelligence of threat intelligence members on the Action Group who from your team is assigned to this thing who are you giving us we need somebody you have a work stream you're going to do that we'll build out this list and then we assign it and we get very secular so the investigation portion happens it's a little bit kind of out in the wind but the investigation is going to

be different could be different logs could be different servers could be we need forensics versus we just need Network Telemetry it doesn't matter we do that and then we come back together as an action group and say hey what have we learned what do we know now well with what we know now we can move back and establish some more work streams because new information means new actions that we can take yeah absolutely and I think you know as we think about hey different types of attacks that you may see okay you've got ransomware you've got you know an account takeover or something like that um you may have Sops for those things and that's fantastic but you're always going

to be finding new information during your investigations that you're going to need to be able to Pivot through so okay you might have something that means you've got a completely different type of attack and you won't necessarily have an sop to be able to account for every single type of attack that you're going to see so that's why it's important to sort of remain flexible keep bringing that new information back into the working group we have the action group and then making those decisions that come out of the new information that you have also for anybody here who who all here Works in IR is someone blowing up a helicopter with an arrow not the perfect explanation of

what we do High tip to Taylor here for finding the gift last night but it is the perfect explanation of this job um all right let's let's keep going so this is the whole model all laid out we can leave it here for a second if anybody wants to take a snapshot or do any of that kind of stuff like you were more than welcome to you might be thinking that we haven't yet covered post-incident activities and we're going to do that with a completely different model in a minute and we actually have time to do that give everybody a minute to take a picture first and then we'll uh we'll jump into the second model that we have that is

going to be a very brief overview because we have a little bit of time that that might be what we come talk to everybody about next time yeah sounds good go ahead cool let's go into it okay so this whole everything we've talked about today is really part of what we call the threat detection and response cycle so this cycle is really intended to explain everything that we need to do on our side to be able to to detect and respond to different threats that are out there so this is something that Sean and I have been have been working through and really trying to understand okay so if we take something if we take a critical

asset in the organization something like that we need to be able to do all of these different things we need to go through each stage essentially uh and be able to cover all of these things and do it well so for example the first stage that we have investigative tool management this basically means we need some sort of tools to be able to use in our investigation this could be a Sim this could be Splunk whatever something like that this could be a number of different things that we absolutely need to have to be able to run our investigation we then need to develop use cases and this is a little bit different from detections but what we're

essentially looking at here is hey can we establish high-level use cases of what we actually care about for something that's going on for example do we actually care about data being lost something being sabotaged do we care about something going to a competitor these types of things we need to understand and how that could actually happen is something that we need to establish and talk about sort of that next stage data architecture hey we need to have the logs and we need to have the data to be able to discover whatever use case that is so if we care about data going out the door how do we actually see that do we need some coverage on our endpoints

do we need application logs what is it that we actually need to be able to see that next stage pretty self-explanatory I think everybody gets this building detections to actually describe those use cases and making sure we have those so our next stage goes into alerting we kind of separate these a little bit out so yes we need to build those detections but we also need to present that in a way that really makes sense for our analysts does it actually Elevate to somebody's eyes that are looking at a pane of glass now that being said this industry changes all the time attacks change all the time you cannot account for a detection for every single bad

thing that happens hence we also include threat hunting in here as a component as a relatively important part of a program to be able to respond more quickly than the detection Engineering Process might take to look for stuff now all of this feeds in to all of it right threat hunting feeds into detection as we get in as we continue moving forward we talk about the incident response and event investigation that all feeds into detection that feeds into data architecture that feeds into everything but that's basically what we were just walked through that so this The Upfront of this whole talk was basically the workflow for stage four now what's really important and this is

where we get to the post-incident activity component mitigation Automation and control recommendations is one of the most important things that gets overlooked by IR people and like investigative people a lot we see things that happen we see risks that other people might not see we then need to take those risks communicate them in a meaningful way to the business to bring proactive change or close those gaps if we don't do that we continue to see the same risks over and over and over again you want to add on to that yeah absolutely and this is something that you know Sean was saying hey we can't just live in a silo an incident response we really have to be

able to work with the rest of our partners that are out there be they in different cyber security areas be it in Human Resources maybe we have issues with something that isn't even technically related like background investigations or something like that if we're dealing with Insider threat type things so anyway just wanted to call that out that this is another place where you can really leverage that action group that you use for operational things now for those sort of mitigation mitigation actions and automation that we want to do you can then they understand the incidents that you're seeing and can really help you throughout the rest of the organization make those changes those Upstream changes that you need to ensure that

you're not seeing uh too much stuff coming down the pipe on the IR area so that's basically the talk and I have to say you must be lucky or something because this is the first time in my life I've given a talk that's like bang on time like just dead on the amount of time it was all the planning we did for this right yeah the one hour last night um so that being said it's also worth saying we don't have like an end slide we totally have time for questions I think right yeah questions okay cool um none of this would have been possible without the conversations that we've had hella some I've had with many of y'all

who are probably in this audience the people that we work with every day except Mark um but none of this would have been possible without them so we have a lot of people to thank and we're not going to name them all but like it's just very important to say that like this is not just us like this is a lot of experience and ideas that kind of built into this absolutely huge thank you to all those folks and this has been you know over the course of of several years we've been thinking through this and and putting some of these things into practice and and we found them really helpful so if there's any elements of

this you know that you all want to take back to your organizations please do at the very least hope it starts some good conversations about how we can build some Partnerships internally and as well get more folks becoming incident commanders and actually working through these things good finish so for questions if possible or if you can I would love if people would come up and like I will like come down here that way they can say something and I'm like if not you can raise your hand if you're not comfortable doing that you just have to repeat the question got it hi question for everyone for you is that um I get this for a 24 7 sock but for an

action team what happens if this happens on Saturday or Sunday we call the action team yeah we call the action team uh so I think that's that's something that's really important and I'm really glad you brought that up actually because you know this Action Group particularly that core the responsible parties in the Action Group do need to be available so we will call people in the middle of the night uh we will call legal in the middle of the night if we need to and we'll wake Folks up we actually prefer it yeah we yeah we'll actually save incidents for early Saturday morning maybe 3 A.M or so and then uh but no that that's actually a really

important part of it right you need to be able especially if you don't have a 24 7 soccer or IR team um to be able to have that Action Group really at the beck and call um of the folks that are working this that is incredibly important um and you have to have that buy-in ahead of time as well so it's not always easy convincing folks to be part of this action group but usually you have some cool examples you can use uh and we you can sucker people like Bunny and doing it but let's be honest it extends past the Action Group to be completely honest right like how many IR teams have full

mitigation control across every possible bad thing that could happen on their Network or do you maybe sometimes need to call somebody that doesn't like to pick up on the weekends like that's just part of the job calling people who don't like to pick up on the weekends like they'll deal that's why it's so important to have this kind of stuff agreed upon across like the higher up leadership of your company because then they can be mad about it but they know it might come what else anywhere please

how do you handle The Incident Commander role going around the globe or around the same lines I can take that one if you want so we genuinely try to like Incident Commander is somebody who is there and available and like can do it we can also hand off that role it's one of the great things about the doing this kind of work it's not hard to hand off that role and say hey you're in charge now or you have a team like mine I'm looking at all of them with disappointment who just don't leave when things happen and just refuse and that's on them not on us yeah so this actually uh speaks to the

importance of making sure that everyone uh in your soccer IR area is really trained to do this uh and gets the experience so if folks haven't done this before hey get them to tag along and Shadow something like that you got to make sure that everybody really can step up into that instant Commander role because hey uh you know you may need to be able to transfer something uh over across you know the weekend something like that um you know or you may have folks pulled off onto multiple incidents or something which which does happen in the IR world the other thing that's worth mentioning is it's really the first couple hours that are critical anyway because that's

when you take your first pass on work streams you figure out what you want to do and then people go off and start working right so until there's new and damning information if there is new and damning information that's when the Action Group gets back together and that's when you can figure out the best course of what to do next so it's very much a like meet plan work meet plan work round and round in a circle all right what else go ahead sir whoever all right oh Elite come over so it sounds like you guys have put this in effect in the last what year or so something like that a couple years so what's the trade-off between the red

tape that comes with bureaucracy when you add in all these different teams and their way they're uh let's just say ability to rapidly respond to something is different than what IR or security teams usually do so what's been your trade-off or how you work through that uh challenge yeah so there's a really important component to the Action Group hey deal with it there's a really important component to the action group that we talked about earlier right that's the responsible parties and support parties now I'll let you talk yeah absolutely so um I guess I guess I'll talk a little bit over Sean here um but yeah I think fundamentally this is why you need to pre-establish the action

group and make sure that you have uh not just a commitment of the responsible parties to actually show up but to actually get the job done and ask and and really make sure they are pushing off uh the current work that they have so we have a lot of times where we have legal counsel or someone else who is dealing with other things that are going on but when we do activate that Inc that Action Group there's an understanding on all those parties that that is the priority to do right now so getting that buy-in ahead of time is incredibly important to do and that's really how we've sort of fought that red tape now

it's not always perfect but generally for those responsible parties that you have um making sure that they understand that oh Bunny wants to jump in this is going to be good so one of the pieces I want to add to this is appropriate air cover one of the red tape Killers is having somebody that's got the facility to talk to the rest of them and make them stand down and that's where you make friends with your HR your legal counsel somebody that can give you that air cover and that's the other piece of the puzzle because without that without having the effective Partners you are going to get bogged down on the red tape foreign

so we had this brilliant plan that we were going to work all this up and we were going to launch a blog post alongside the talk and we're going to do all that and then we wrote the talk last night and have none of that um that being said we are planning on doing all of that when this recording goes live so like we should have companion stuff and like the actual workflows and all that kind of stuff around that time presuming that legal approves or I decide not to listen all right uh we probably got time for one more anybody please

how do we train people and how often do we exercise for the clear roles and responsibilities you want to take it yeah all the time um hopefully we're not putting this into practice all the time uh that's a that's a really good way to to train Folks up but I think one of the core things and I think this is something I really want to call Sean out for doing a really amazing job with is sort of as we started this Sean is a veteran Incident Commander here he really took that role of The Incident Commander to sort of LED as the leader in the organization said This is how we do this and so we built a lot of that

stuff and modeled a lot of this stuff off of how Shauna's do it has done this and as he's put some of this stuff into practice we've brought a lot of the more Junior employees along to sort of observe that I think there's really nothing better to do than to really understand how this stuff gets put into practice and activated so that's really one of the ways is making sure that our our Junior employees are here actually see Sean or really any uh senior person in your organization put this stuff into practice uh the other piece is that yes we have the specific processes all documented and outlined mind internally as well so that folks understand this

and are able to essentially follow along with what this what the requirements of that role specifically are in terms of regularly Drilling and training this is something that we that we do probably not as often as we want to um I think regular drill of this sort of stuff is is actually incredibly important it's just sometimes finding time in an IR shop to do that which we we find a surprising amount of time now think about tabletop programs and all that kind of stuff like there's there's quite a bit Yeah so one of the things speaking of tabletop programs that we actually think is is pretty important to do as well is to make sure that you have

tabletops with your Executives using this model or whatever that you have so when you come up to some of those big time decisions that need to be made during an incident you do need Executive cover for that as bunny was talking about earlier making sure that your objectives your executives are roped in Taylor they're telling you to shut up yeah I'm getting pulled off the stage I thought that you're going to play the Oscar music to get me off stage was that no you'll have to come up and drag them trust me all right all right thanks everybody so much I appreciate it I'll be around to talk [Music]