BSidesATL 2018 - Ozymandius and InfoSec RockStars (George Roberson)

the title is a little bit elusive what we're really talking about today but I want to talk about is basically humans as a single point of failure in d are busy people and so the hell you made these reference come back around but the info Rex set rockstars thing basically about irreplaceable paper so we'll get back to that real quick just about me in about there did you see they don't we spun the question raucous reputation defender I'm the information security architecture unite repeating things all thoughts and opinions are mine don't have that on slides anywhere that has to be mentioned so I've got my Twitter handle up there I usually use that to kind of follow the community not that

all that stuff I'll try to get better about that I know it's one thing to always one contribute which is the reason up here so I've got about 17 years in tech InfoSec or some blend of the two kind of came up organically through engineering their training working with mostly supporting financial institutions or consultants working in with financial institutions so a few of my favorite things like she's won a national couple things I'm a little bit documentation freak I enjoyed doing documentation and doing well maintain documentation it's helped me in my life I know that's not a very fun thing but this talks let them cover a lot of fun things but it's good stuff and it's

tough not to talk about nothing just babbling like regular guys god bless everybody that works in unregulated shops having that back to be able to take the management when I have a security control I won't hit me man so I'll have to argue it on my own but say this is a good idea we need to do this but these guys might shut us down we don't it's always been helpful working without that crutch changes things quite a bit so about this talk standing on the shoulders the general when I thought about first submitting this tall just have to be such chat nugget I've heard several good things maybe I've got something fun site and

then I jumped on the soil but this one had a thing and it changed what I want to say because in information security we work as risk identify herbs in a lot of especially blue team you know you're there to identify risks and close those gaps so my mom being positive and optimistic immediately went what happens when you lose their jobs I mean metaphor of the standing on the shoulders of giants is these are the people that got me here but what about the actual people that we rely on do we figure them in do we think about how they help us and do we identify them as risks that's something that can bring your company

then known as an insider threat but that's just somebody that you can depended on so I'm gonna go through three quick things I didn't find the risks recovering from loss and I've got recovering it there but I'll get more to that but really what consequences you're going to deal with when it happens because you know the recovery is there's not really a recovery from it and then mitigating the probability which is the proactive steps there at the end and it's again I'm just like saying we all kind of know these things but we don't talk about them so we don't really consider them and put them into our processes first a practical example so several years ago we lost a network

engineer senior network engineer at longtime employee cornerstone of the department very cool guy smart tireless ferocious when he came to maintaining that time but mid early 50s she was a runner healthy super fast it was always fun to go to a half athon or 5k with him just to see the other guys in his age bracket your faces draw because you know they knew they're gonna get put down soon as he shows up so I was in a conversation with him in the hall one day I was asking about its running and he mentioned they hadn't been able to leg lift because his legs were hurting and so just a little bit after that he

started missing work just sick and then we got an email from his manager they included the word aggressive which is not something you want to hear so it was weeks I mean it was no time and he'd been pretty village about maintaining some of the things we'll talk about so through no fault of his own his backup his primary Becca had just been went on better pastures basically it found with other opportunities and so he was coming a gap between replacing that talent so that left us in the nearest is fine learned a lot of lessons from it I'd like to say everybody on my team did a fantastic job dealing with it but again you'll learn a lot going

through something like that we'll start off with some massages if I think much then I'll a crazy story guys so just going back to the examination handbook there you should never assume that critical personnel including senior management are going to be immediately available during a disruption so if you're going to position where you're missing those key personnel and you have a deal or a basic piece of REO is that figured into the plan I figured in whether they're on the road whether they're in the hospital whether they have to deal with their own families those kind of situations can take those people away from you and really complicated humans once they put up there humans aren't

highly available it's not like we have instant replacements and sometimes we just aren't available at all so looking into that at risk identifiers we need to figure out how can we look at our organizations I mean we can proactively plan around this but we can look at our organizations we can look at our cross and we can have NFI those risk factors what you're looking for is those people and organizational trends that tend to pile up responsibilities in such a way that they can't be replaced so I broke it into two if you've got organizational factors so one thing and we'll get back to there's a the causes of these and people's responses to these but the rate

of growth of your company or your organization for your business that you're just even adding a customer base and the volume of works going up not so much change it doesn't leave room I mean you know your days full you're assuming we paid for a full day so you're doing a full day's work so as that volume increases if you're not having kids and adding in stretching other days or shifting workloads so the cross training documentation planning around BC PDR can be done you're going to end up with more of a single point failures the next one opens good turn more of an under managed acquisition so if you don't have the project management structure in place so

that when new technologies are brought in or acquisitions are made or where you've got shadow IT departments behind their own stuff if you don't have a project manager or management team then it's saying okay we need that's not an ownership we need to plan back up we need to make sure our vendor management processes involve this if you don't have all those things a lot of stuff can sneak through and that stuff's going to stick to qualified people that you need doing other things and the last one it's kind of weird one but working with some people in organizations I have high retention so if you have a cultural thing in your company or the geography

of it or any other factor you have people quite stick around I mean it just could be a really good place to work longtime employees tend to acquire additional dues they tend to acquire a lot of institutional knowledge that makes them in value so conversely turnovers good way to highlight weakness if you lose somebody you know you've got a problem if you've got a higher turn over you know what needs to be hand around because you've done it so many times it's always a good place to look is the people that's been there a long time to inventory what they do and we'll get back to that near to mitigation so the human factor and I was thinking

through this is it's how we spot these people how do we spot the people that are just watch said their pleasures are invaluable to us and there's really three risk factors that can hand out stuff so when there are couple different things to call talk about you know maybe put a superhero reference or two in here but really nothing to fit so just want to say I know all these people are combinations these people are people that are more than these thing and so it does come from experience but you all know headway so first I've got the Golden Week and this is pretty much as someone who everybody likes to work with so they've been assigned a duty or

something they're supporting something they're the guy that they go to for this particular type against in everybody gets used to working for working with them so even if you assign them a backup that backup never has to pick up the phone because their mail columns they can I get such and stuff or you know basically this they'll always email try to certain anything you set up so what happens is their secondary is the neverending cross derivative and so if you lose this person I mean it's obvious but the interesting thing about this too is if you promote this person you promote this person you expect him to a new job all that crud is gonna fall

along with them because the people that's been working with them all that time are still going to want to to them so that creates one problem the next one is the juggernaut so basically found that has everybody most people need sleep food family time you know these some things there are some people that I've worked with that don't seem to they will work hours upon hours they can work weekends if there has to do something they won't question it then we'll balance it out so what problem is this great you can't replace that person with one person everybody else that has expectations of work-life balance won't come in can't do that job so and you

don't always know it because you just kind of you ask them to do something they do it just everybody some of them complain and do it anyway some of them don't complain and you just never know if they're gonna break it might be yours it might be when they're working for you and might be working for somebody else the problem is it's when they go away you come on so last when I put the polymath now this is this is somebody with a broad background especially you know talking on the technology side so somebody's but I think if they're good at everything just give them all the things so if you've got a guy that knows web in programming

and database and works with people and knows how to manage well here's a project you want to give your products to your best guy if I'm in with the project management and control then if you don't really quantify what your important things are you've ended up giving them all to one person and nobody can cover them unless you've got to which is unlikely what happens up happening is is he's got a bunch of different systems with different skill sets to support him so it ends up being a lot of junior admins or sysadmin jury best people or you know dude you're handling end up supporting off of one thing this thing or that thing so when

he goes away you've got a whole bunch of junior admins that become primaries on something that's important here part and then you don't have the resources across truck and we'll get into that some of the consequences afterwards but they don't immediately know how to hand that off and it takes a lot of planning to recovery so identifying these one good way to get these things to show up these type of risk packing shorter it's there testing everybody doesn't they are testing out the hope at some senator been which product is but I mean it's the first question is when you have it in Belize to say let's do our deer test and let's look at your BCP plan if it's

only got one name on it you people maybe people got a problem and it happens and that's a that's a simple thing you just test fails pick a racehorse trainer retention of ice ball will be busy but that right there into this problem the second one again it's just your PCP point you might have a secondary assign everybody might be used to going to the primary so that's one of those things especially in Incident Response as the legs to just security and as well as the PCP in the art when the incident occurs do they know who the second areas they know how to contact them does those first responders are going to be contact

center people a lot of times so they might not have that so basically if you start your PCP testing point to do that first responder was and say this is happy what do we do about it and install is adjustable every test the last one is going to be more common so you have a secondary but they're not gonna be as fast the normal B is good they might have to Google some stuff they might have to read through the procedure so if you've done your dr testing you're basically testing and you've established a recovery time objective based off that with the primary test it again takes to begin with a secondary figure out what that

time is now again you can retrain them try to make them faster no way has time for that so another important thing to remember is you can adjust so far TOS unless you're in a business that's so highly regulated that you have I mean this one I think if it's absolutely has to be uh it absolutely has to be up in an hour you're going to have to fix that but if it doesn't you know it could be that management can understand that you can say if such-and-such is available so we get to someone an hour if he's not it'll take us day it's all about setting expectations now you have to give your regulator the longer time but like I

said what you're doing is you want management to have a clear expectation of what you can accomplish with which resources so that's all said to say it's okay to fail on these that's how we learned that's how we adjust and identify these risks in response so I don't have time for questions in this talk stuff with this next n because I've talked with people about this a lot and the first thing they pop back with is there's no time they're giving us more stuff we can't be cross-training to the level where we actually have good DRB CP we can't have this kind of cutter because we can barely documenter stuff all that that's all fine it happens

everybody that's what they say so

it's a better is it all battery batteries did all right I'll just a little he'll pop those out I'll just get a little louder until he gets it done so when the response comes back that you know there's a lot of frustrated technicians a lot a lot of people that I talked to just that the resources aren't there to accomplish that that kind of full time coverage get those expectations where they want to have so first thing is it's kind of calm down it's not hard response a bit well I say unless there's management in this room and executive management the truth it's not our responsibility to fix this we identify the risk we offer solutions we

offer mitigation strategies but at the end of the day it's our job to look at that risk and setup and management expectation about it the staffing decisions that they make to fix or not to fix this that's a risk decision they didn't take look at these numbers and say well if if if Joe is unavailable we're down until he comes back if he's quit or hurt we're down until he comes back that's it they can say that's fine not having that say um so flex it not having backups people or data it's an option it's not a good option it's not something you should do but it's one of those things it's management's job for that you know not being insured

that's not - like I said it's at the end of the day it's their business burned down you just have to set expectations so when something when the chute drops they know what to expect looks at this the this slide here you know you can't get dressed the park back online with left in snickering you know when those characters we've seen that movie that's not the first time the gray said that to John and so the look of surprise I'm drawing his face shouldn't be there because you know this has been explained before they build that Park and they have when I take it so you know it's one of those things set the expectation

so here's where it gets bad consequences what happens and this was this section was originally what do we do what do we do with this what can we do how do we respond and the answer is you don't really have too many choices so there's not planned appropriately for this it's not so much what are you gonna do is here's what's going to happen if you've lost a pillar that has a particular product knowledge or expertise you're gonna have coloring consultants you're going into cullinan resellers or that company itself and really not a good solution but it's what you have to do and it's actually the better than the second one which you've lost someone who is a kind

of a cornerstone institutional knowledge where you've got a lot of weird systems or home-built stuff and so now what choice if there is a former employee that has been for somewhere you might can lure them back they're gonna have a knowledge gap of the time but it's possible and it can be done and they'll grow back into it and you're a really good solution is if you've lost your giant to a consulting company that's pretty easy because they don't have to tell the company and give them back you're paying you know three times what you work for for part-time people but you know the system's data third down there except it's not everybody can do

this it depends on your business model but maybe maybe you were building a particular application that product no longer exists if you don't have somebody to support it you don't want a patient results bring somebody in that knows the tag knows the code you can always drop it I mean that's cost it's a long story curves and reassigning is this to resource I put that out there because that's likely what you're gonna do but it's not a real solution because that person's doing a job that person is no longer doing that job at least in some percentage or fashion so you're really just moving the pain points between people so recovery steps this this is just a

real quick one when this happened again it's not recovery you're going to do one of those four things but what you need to do is you need to treat it like you just had an incident or D'Arcy situation or PCP you need to sit down and have a lessons learned do a full inventory what got dropped when that person gave notice or otherwise left you're gonna have to educate secondaries if you have handed stuff off you might have handed stuff off to junior admins that aren't familiar with the project management are familiar with BCP planning don't know that they need a backup now and that will just like set push your problems further than probe and you need to

document it as an incident and the reason for that is when something like this happens there's a cost again management made a decision something happened the businesses have been impacted there's a cost associated with that write it up and provide a tool that will help justify bring down the road as first how to fix it so quick notes on mitigation it really comes down good work which is in the slides still you have to properly equip the backups this will happen it's you can leave out of here and say okay now we've got backup so we've got good documentation the key is the organization isn't enough you have to have job rotation I say cross

train cross trainings meets the devil to you tell you what I do you go back to your job doesn't work if actually we want to be prepared for these type of events you have to do job rotation that means weekly monthly quarterly whatever is appropriate annually is not gonna be appropriate for many processes but the reason you do this because it solves a couple of things first off documentation goes stale so if you're going over to the training and you you're not driving and showing them they're having to do it you're instantly I think your documentation because they're gonna run into snags they're gonna run into systems have been replaced they're gonna run into people

that are no longer in the contact list all kinds of things so you're constantly and continuously updating documentation the other thing is access issues you run into this lot maybe their permissions have changed on the system you know what happens if you're not they're trying to do it and then under pressures you so that it slows the entire process down and pushes out your recovery time Jacob in addition think they might be missing software might be browser incompatibilities again they might be trying to contact people that aren't therap or don't know how to contact it it might just say there might be a job role with the documentation and they that is or that role might not

exist anymore so going through it with a proper job rotation is really the only way to ensure that you actually have is asked to recover this BCP capabilities as first people so the other thing I put here is that doesn't mean that they need to 100% do everything that you can do and they don't have to do it tomorrow it can be continuous or you can continuously improve rather than trying to get it done immediately the other thing is is made it expectations if they expect somebody to be fully functional what is it fully fully armed and operational but they don't care if there's you know they don't care if it looks pretty they just need to get done

in a certain amount time they understand that there's a difference when this one is doing it that one as long as it does the job it doesn't matter but again management expectations so there's a couple quick notes my last few minutes here some personal applications and I just say this actually goes back to the first talk there are certain ways we can apply this in July doesn't make getting this right attractive so first off letting go for ourselves and the people that we counsel it's important to think do you want to be doing what do you want to be that person who's absolutely replaceable so you don't get the sleep you don't get the go on vacation you don't ever get to

do those things if you think you're going to duck you into yourself out of a job there's not enough of us for that it's not going to happen so that's just quick picture that's the when I came out of infrastructure and was able to train with a replacement and move information security I was able to take the first two-week vacation of my life without ever logging in so that's that's actually made so we crawled up along the coastal towns in New England for two solid weeks I checked email as it never lofty and never touched the big P so does that mean the company don't need me no it just means everything good either wait somebody else knew how to do it and

so if I was actually taken down by something the same result thing but it's good for me it's good to get the cork off balance makes you a better employee and the second thing which goes back to what she said it's noir you want to advance the profession again for your own benefit if you can't find people to train if they don't have the skills to cover for you if you don't trust them if you're the golden boy to yourself and you don't trust anybody else to do your job that's a problem you'll end up the same boat so what you do is we sell for us as information security but you you recruit a speaker you tell people have

you tried this would you like to do this do you think you'd be good at this and like she said you mentor and once you do that what's that you've got people who can replace you now that's the plan you might be in a job where you don't want to be replaced but do this anyway because again it's competent service to your employer because something might happen to you outside your control but the other thing is you're helping this person you might train them to be your replacement they might never replace you but now they have opportunities and skills that will further their career allow them to pivot the things that them at each window so back to the table

buzz Amanda all my favorite poems Brian can't go through all that the Bryan Cranston does a great reading of it on YouTube for one of the seasons of Breaking Bad mind it's good basically it boils down to just the point that it doesn't matter what systems you build it doesn't matter what you write down about them all that means nothing unless you have people that can carry it on after you if you don't pay attention to the people it's just it's all gonna fall in room this runs forever so that is the end appreciate town [Applause]