SOC Staffing and Scheduling: Justifying Headcount and Meeting the Mission

appreciate it thanks can you guys hear me okay yeah barely can we take it up you good now good thumbs up kinda okay cool this is our agenda for today we won't beat that up uh this is me really quickly like if you're asking why do I talk about this I built three or four socks now I was recently at Microsoft I'm at Oracle now and so we've built a sock and we've grown socks a few different ways I've done a sock from scratch with Dell securix and Key Bank and I worked in this and I'm also a soldier I was soldier in the United States Army Reserve in the Cyber protection Center out here in moon for about nine years so

that's my background I would like to highlight that the fun fact at the bottom Ohio is the greatest State we can talk about it later we can get into a debate boo I understood yeah that's okay so what is a sock um most of you probably have an idea in this of a sock in your head right like you think I never secured Operation Center is it's tier one tier two tier three and this is how it works but it's actually if you look at cobit nist ISO anything like that they will not Define a sock it's not there try it Google it look for it what they actually Define is your incident handling process so you

have this tier one tier two tier three mentality and it's actually broken into uh it's the mindset that's a task org what you're seeing on your guys's stuff and it's actually an incident handling process so the way to do this is to understand your mission and the only way to understand your mission is to understand your hires Mission so I'm an army guy so we look at higher headquarters Mission we break it down to our mission that's how we do it and it keeps going all the way down so if you're in a big organization you probably have a dedicated security operations center right tier one tier two tier three you may have a smaller if you're in a

smaller org you might have just one or two people and they rotate through that'll be on the next slide but you need to understand what your left and right limits are in terms of mission because there's going to be teams out there that do pieces of let's say IAM or they do offsec or they do uh digital forensics but at a certain level when the the hits the fan they're all incident handlers right because they all Belong To The Blue Team so you need to know your mission your organization's Mission uh and then you also need to know your company's Mission and how those all tie together it'll allow you to like drive your mission all

the way through because you'll be able to say our mission says this and so it's important so what does your stock look like today this is what a lot of stocks look like this is uh tier one tier two tier three we also see some socks that have a disjointed or like they share resources that happens all the time um and one of the things that you need to understand is like when do you click that in when do you start sharing resources your triage or tier one like typically that's what people think of a sock they think of this like 12 Man team 24x7 operations an mssp contractors all involved and that's it but it

actually goes all the way up to the tier three you can go all the way up to your seesaw your CSO can be part of your sock but it's really your instant handling process that you need to think about there is also a thing called a round robin which was really popular in the like the early arts and in the 90s and it's still still prevalent today if you're in a smaller company you'll take a cell phone and you'll pass it around you're like hey you do this week and the next week you got it and week after that you got it that's awesome if that's what you have to do and you're making it work

great but I'll tell you this way if you probably don't have the experience to like triage everything by yourself you're always going to end up bringing in more resources if you do it that way so just something to think about like hey maybe we need to dedicate some people to that and this is the kind of talk that'll get you there so know your objectives and your constraints there's a lot of stuff on this slide I'm only going to focus on three three or four of these things one is the internal and external customer slas you probably have contractual requirements whether that be through regulatory reasons or maybe you have a contractual obligation to a customer

that you set just know where those slas are because if you don't know them you won't be able to like build your other metrics off of them this is a really metrics driven talk to Define how you guys are going to staff so super important the other one there on the right is your budget if you wonder that you said hey I need 12 people but I only have six the budgets why that is and you need to find some other ways to work around it and we'll talk about how you can justify getting more head count or just explaining the leadership what happens if you don't have the head count that goes with it in a couple seconds

but the real big thing is uh I'm I was in ic before I was a manager I'm a director now but I'm a big fan of work life balance and your employees are if your ICS in the group here like you want your leadership think about work-life balance all the time like the worst thing for me is to like how do I get more time to myself self and my family I would work so we always trying to hone that in and find the balance and so something we want to touch on too we'll go into depth here in a second this is a lot of text to read but uh the biggest things to look at here is how

many cases do you get in a day what's the severity in those cases how is your sock going to connect the network and work those cases this is important without talking about things like on-call or if I follow the Sun or mssps and then how do you set up your schedule so most commonly you know we'll talk about five by eight I'm sorry four by eight sorry five by eight four tens and two two three and other stuff here in a second but um I have some slides on that but really quickly those are the the key points here to talk about when you're building your staff plan and then what do you do during an amazing major incident you

probably have an incident handling policy at your company it's written now it's stuffed in a box someplace and you're like ah we will use that when it's a big deal but you hardly ever test it when I find companies like they don't do a walk through they don't do a simulation or parallel tests they've never gone through it and they haven't said like really take your sock out of the picture if you haven't done a coupe or done any type of Disaster Recovery like just take your sock sock out of the pitcher take your tier one and put them in a box and be like nope you don't participate how does it work now test that out that's important to

highlight too so let's just talk about how do we get those numbers there's two ways we can get the the case coverage metrics and we can take your cases now if you don't have a case management system like jira servicenow RSA whatever your tooling is you can take these from your your IDs you can take them straight from your EDR solution you can gather these up a few different ways but most people work a 40-hour work week or they try to and so that gives us uh 2400 minutes in a work week and so we need to divide out that time so most of you are probably aware that you don't work 100 of the time at

work if you do let me know how you do it I'm interested so a 60 is actually realistic 75 is a really easy way to justify it if you need to sell it but say hey I have four analysts that's 1800 minutes times four and then divide that by the number of cases you have in a week that's like you know 15 minutes to triage and work every case to closure does that sound realistic some cases yeah it does it's super easy some cases it's not possible so think about that when you're just fine that one number won't stand alone you need other numbers so we're gonna keep going so the other way to do this

is to take the number of cases you have and then times them by the median TTR that's total time to resolve for me but it's a total time to close for you maybe whatever your metric might be if that doesn't work out I think that number whatever that number is and then divide it by the by uh 1500 or 15 I'm Sorry by 1800 which is the amount of work you're going to do in a week and you end up with the amount of analysts you need to work your current caseload if that number is less than you currently have how do you do it that's the question I would ask like that means you're either Auto closing a

bunch of cases which is totally fine just remove those numbers and you get a more realistic number or you're overworked in which case you're letting things drop to the floor which which sometimes comes out of the budget you have to do you have to prioritize when we're talking about when do we work this is important too uh we talk we'll talk another couple slides about like the the scheduling methods um how do you decide when do you work if you if you're running an incident handling team when do your analysts work traditionally in America right like nine to five that's a really common one everyone says oh we work nine to five that's a really common everyone wants

that schedule but is that best for your company what if your company's in Pittsburgh sorry your company is uh in the west coast but you actually have your sock on the East Coast does it make sense to have east coast hours nine to five it doesn't keep West Coast hours so when you have a global company you have to like think about this a different way what if you're that you've outsourced your tier one and you put them in India or in Romania and then you look back and you're like well we all work PST hours so then the hours need to look different so kind of think about that and really easy to get these graphs

by the way this histogram is just to take your off logs and then just count by a unique user by day and then just graph it about it by hours by minute by hours and you'll get down to this like type of chart and you'll be able to say this is how many people are logging in on a regular basis and it gives you the the idea of how many uh when the activity in your organization actually is picking up another thing to touch on real quick is when you're thinking about on-call capabilities there's technology that's enabled that will enable you to do that right you probably have pager Duty servicenow some type of integration

some companies don't have that you may need to invest in that technology to do this that's okay also when you're talking about on-call if you're moving to an on-call model we'll touch on social tax here in a second but if you're moving to an on-call model for like your company's growing or you're hiring more people you want to be more advanced than security one of the things you can't do is change the deal on people without communicating so if I work a nine to five and you come in and say hey Chris now you need to work a uh you need to work midnight to 8 A.M and you're oh by the way you're going to be

on call and you never did that before you need to socialize that with me and sometimes for some people that's not going to work so then you get this rub of like are they like is this going to work out for both parties the organization and the employee so this is just a really quick one for you on-call coverage if your SLA Gap if you're SLA for your detections is bigger than your Gap you're not going to need SLA this should be common sense right but some people don't think of it like this don't think of it like break it down and that's if you have an on-call capability you don't worry about that so

24x7 Ops is really common if you're trying to follow the Sun or do something like that but one of the things you may not think of is that some people have UB networks or a or Gap networks so they can't connect to you don't have an on-call method there there's no way to remote in so now you have to think like hey I have to connect so I can't have that 16x7 I actually really do need a staff at 24x17 to do incident handling so how does that work so always think about connectivity zero trust and your how you're going to VPN in and get to your guys's tickets a lot of things out there nowadays are awesome teams has

integration slack has integration we have chat Ops that does awesome stuff so how does that work for you in on-call is important scheduling uh so like I said most people work five by eight so I'll tell you that's 260 work days a year I'm a big fan of 410s it's that sweet spot in between and it's 208 work days a year that sounds I mean just really clear that's that's 50 days off more for every employee that's great that's that's awesome but it is striking a balance that is a 10 hour work day most people will say after six hours they're not being super productive at least that's the metric they use to like try to justify having a shorter work

week you can go to a small like if you want even fewer days you can go to something called Pitman or 223 or Panama schedule which I'm sure people have heard about it's a two on two off three on two off two on three off that sounds confusing there's a chart okay I I promise you like but you can do it if you want um I will tell you this it all goes to that social tax if you want to switch your team from a five by uh five by eight to a 410 just socialize it make sure it works 410 sounds really sexy to a lot of people like oh yeah four days a

week that's great but some people have to put their kids on and off the bus some people have to to do other things just keep it in mind when you do that you like you may have not signed up for that for everybody and you may have to pay the social tax of normalizing it with your teams and uh you know it's work life balance so try to get it right Staffing is important and this will enable you to do that if you plan this outright you'll know exactly how many people you need when you're doing Staffing and it'll allow you to say I need XYZ people uh so I can actually get this done

follow the sun uh it's just really straightforward if you want a 24x7 sock you don't want people to work at night you want to keep that work-life balance really cool you think working during the day is the best and that's when people are most productive follow the sun works it's also probably cheaper for you if you can do follow the sun this works just look at the cost of living it's broken down right there you know by country this is this is a way to do it not saying do it this way because everyone has their own requirements everything here is like a how do you want to do it it depends talk um cons here there is cultural barriers

there's going to be times when you are Community I have teams that work in the UK and you think that the UK and America are as close as you can get in terms of cultural barriers nope tons tons of things that are different and it happens all the time like for example it's a to give notice in the U.S it's typically two weeks right in the UK it can go anywhere from a month to three months so how do you staff a team like that it takes a long time uh loss of connectivity uh continuity during handoffs if you have these handoffs and they're contingent on hours across the world things happen another thing to think about is holidays for

example like holidays match up in different places how do you do coverage on holidays with different countries it's a lot harder to schedule when you're not keeping on top of it so just adds a layer of complexity if you try to do follow the Sun and then there's mssps or Contracting you just buy a service like hey I want you're going to monitor this for XYZ that's going to work sure it works they you get the leverage all the new technology Stacks or your own internal stack depending on how you do it there are some good things there just keep in mind too that you're relying on an external party that's a factor there is contractual challenges

that typically cost just as much if not more to use an mssp than Staffing your own people but this gets into this core versus context thanks this core versus context and one of the things there is like if your company does makes uh I know printers you probably don't have a lot of core in cyber security maybe you do maybe you don't so contextually it makes a lot of sense to Outsource your cyber security to another company and that just makes a lot of sense so let's talk about Staffing and Reporting the degradation this happens all all the time I see it so like so we use an example up here that it's a 24x17

that uses four tens they're the you have to have an on-call I'm sorry where on call is not an option and they have a four hour SLA so just have this chart handy just to bust out something like this for your team say at 12 people we are operating on tier one completely efficient we have all the things covered all the shifts are covered with fault tolerance we're good like someone can take a vacation a birthday take a day off it's all good at 7 to 11 one of those shifts doesn't have fault tolerance okay doesn't matter which one it is in yellow let me go to Red and it says anything under six you've lost coverage

somewhere if you have that four hour SLA you're missing coverage you're missing the mark as a company what I will see happen all the time is they'll say hey I only have 10 people for head count but I need 12. but they say but we only have 10 people so I'm green I've hired all my people we're green they're actually they're actually yellow other red I've seen it go down as that far too oh darling all they have a six head count but I need 12. all the shifts are covered so I'm good no that what that ends up being is this so like hey I need you to come in on Friday man someone has a vacation they're taking it

off all right someone's sick I need you to come in and do some work or um hey you know this is an incident I need you to come in and work on your day off no you need a staff so that people can actually take time off and so this is what uh if you end up doing those watermelon metrics which is it's green on the outside and red on the inside this is what ends up happening and and we're almost out of time here these are things I wish we had more time to talk about I'm trying to get in and out of here so you guys can get this talking get on the move but training is

important burnout's important and imposter syndrome is important I will tell you this if you're in this talk right now and you think I'd have imposter syndrome there's the Dunning Kruger curve in the bottom right probably really hard to see for you guys in the back but uh if you're if you're here you're in the valley of uh at least if you're not on the far right you're in the valley of Despair which is means you know you don't know something then you're here to learn so thanks for coming next year I hope you get the chance to get your own talk because it's really not that hard to put a talk together especially a 20 minute talk

please come out next year and give a talk to yourself any questions

uh all right thanks great talk uh do you have any recommendations for tools to help manage a schedule to manage what a schedule yeah so actually you can use so this is the way I do it um which is you can use uh co-linear and War tools which is a python module and you can put in all your constraints and then it'll stick you can pay all your employees preferences in and it'll dump out your schedule for you yeah so I wrote it I can try to share it but welcome to have at it there's other professional tools you can pay for that'll do it too so I just didn't want to pay so any more questions

cool thanks everyone I appreciate it thank you