Between a SOC and a Hard Place

hey everybody everybody want to try the mics again check one two dustin why you so quiet what's that you're a little a little quiet a little quiet all right there we go that's better you guys hear me out there yeah can you hear me now i only see two thumbs that's it i see thumbs i i i see two thumbs okay i see the doors closing so i think that's our cue to go okay i hope everybody so far has had a really good time at b-sides charm 2018. there we go there it is i really like this event um it's been a lot of fun i've had a lot of fun here so far

so everybody welcome to between a sock and a hard place we're going to start with some introductions um just so that you guys get a chance to know the people up here on this panel a little bit first up we have jp over here that's a much older picture it's not that old the kids will strip we're still teenagers and then this happened so it's under your shoulder been around for a while i was just looking this morning and realized that my first infosec job the first thing i did was in 1986 implementing uh army regulation 383 i was four i wasn't born yeah so yeah it's a little while ago uh while i was on active duty stationed

at an army base that no longer exists uh in northern germany uh making sure in the back then the army regulation about uh was called automated data processing security was all about personnel security and it was uh it was aligned towards uh very similar to the r the nuclear surety program so we had to go around making sure people who touched automated systems had their medical records flagged and we got notified if they got any weird medications prescribed nothing about network security or anything like that long time ago and then it involved cpm machines anybody ever do any work on a cpm all right there we go yeah i'm that old all right next so what are you doing

what are you doing now uh right now i am working uh in a dod space uh kind of help helping to mentor a uh special mission sock um doing it as a government contractor yes i'm one of those um so uh and helping to mentor and making sure that you know folks get trained get you know we keep innovative and keeping on top of things does that help that does thank you that's good all right next up [ __ ] stick we have oh oh that guy so my name is andrew marini i've been working in the cyber security world now for what like nine ish 10 years something around there i ish went to school right out of college

right out of high school for a network administration degree up in pittsburgh that's where i'm from originally but i went to school with network administrator because i wanted to be one of those server monkey guys we're going to serve rooms you know running cables quickly realized that everyone wants experience but refuse to give refuses to give it to you so met sean in college been friends for a long time and he ended up going down the security side of things so i called him up one day i was like hey can you get me into you know where you're working at which was mssp he got me the interview the interview was quite simple it was just

like where do you put a router and where do you put a firewall in a network diagram and they're like what's port 53 and 80. okay you have a job so coming out of the coming out of the gate and as a junior level analyst worked my way up coming moving down here from pittsburgh uh worked in everything from government socks to private sector socks to an mssp and most recently i stepped into the world of being a senior engineer and i got married so right now i'm a another dod contractor working as a senior security engineer on a couple security tools thank you dustin i think i have the best picture i i think so

it has beer that's full tilt beer if you know anything about full tilt uh they're a brewery out of baltimore right no i check them out all right that's the caveat i've been doing stuff in socks for a long time i started off as an analyst and then uh i had a couple customers ask me to build a sock for them so that's kind of where i'm at now i go around i'm also a contractor building socks for government and industry clients and my wife's here i want to give a shout out it's her first conference uh so here [Applause] thank you thank you and in this case i'm definitely saving the worst for last

that's me um i am the dreaded manager at this point um i used to do breach consulting really what i want to take time on is not to focus about myself for my nine years but i want to give a shout out to a hacker space over in severn since this is besides charm on allocated space i host and manage the infosec classes there every other thursday completely free so for anybody who's interested in learning some stuff we have a bunch of different instructors come in and talk i also have a twitter at understudy77 and i really suck at twitter so that's worth knowing so now that's out of the way we can kind of get into it a

little bit so let's get started what is a sock anyway andy oh well sorry so go there i mean i guess we can go over there so i saw security operations center i mean you have everything from you have your government socks you have your private sector socks and then you have your mssp socks so i think we're going to go down like so you a stock in general right it's a a group a group of teams that is designated to protect something if it's a government's uh government stock it's we'll get there yeah slow down sorry i apologize for going quick because i didn't know i was presenting this slide i like putting people on the spot well

thank you thanks for the heads up but yeah so security operation center a group of people we're trying to protect the network in some way shape or form with some oversight maybe perfect and we have one big disclaimer here not all operation centers are created equally analyst capabilities these things vary wildly from place to place including what you're going to see on this next slide which is what andy was starting to talk about the three common types of socks highly oversimplified first up we have government i think that's my turn it is so government socks have uh have the responsibility of ensuring protection of national security assets um sometimes it's nationals not always national security but national assets

primarily i mean department of agriculture anybody here depart shout out for department yeah um they they're protecting their stuff too so it's it's not necessarily when people think of national security they generally think of the classified realm and things like that but you know all government agencies whether it's federal or state or local have some responsibility protecting the data and protecting the mission that they're charged to do uh having worked for nine years in the commercial world it's a lot different because in the commercial world it's profit and loss making sure in you know shareholder value value but in the government world we're really working on ensuring um you know the national assets and that's why next

up is private sector yeah so private sector typically when you're working in a private sector shock when government stocks are just an additional government entity while in your in the private sector what's a company's primary goal to make money and as a in a private sector sock you're typically going to run into things like you're protecting ip you know you know anything that you know data loss prevention stuff like that so you don't want any of the company secrets getting out but the difference between government stocks and private sector is you aren't making the private sector company any money you're just a big red mark on their yearly budget so that's always a big concern so

just you're there protecting their information so it doesn't get leaked out to somebody else trying to steal a prototype or whatever but you're always looked at as you're costing me money you always need to have not always i mean companies that have been burned will will realize that maybe perhaps it's a good idea then eventually they do but typically you're always looked at it's like this is costing us money how do we make it cheaper you get about a three month window that i've seen from consulting where if you get burned you got about three months to request budget and requisition new services before everybody up high forgets about it and then they don't care anymore

so lastly we have the managed security space the mssp managed security service providers so that's kind of been my space for the last several years both in consulting and in sock work that's really i'm company a and i don't want to hire a team so i'm going to go to company b and i'm going to pay you usually a smaller amount of money than if i were to hire my own team and i'm going to outsource all of my stuff to you you're going to be my soccer you're going to be my consultant so you're going to do this or that for me typically those places work on lower money but they're a services specific

company so their revenue is from providing security for other people so they tend to be easier to get into they tend to staff a lot even if they do tend to have a lot of turnover but they tend to be really good places to be from that perspective the downside of something like that is you're always going to be about an inch deep and a mile wide rather than in private sector or a specific government space where you're going a mile deep so i would say that the mssp done right is actually a like brings a lot of value for the small and mid-market companies that can't afford to do it on their own or or whatever um but

there's always the old saying you know and you understand when you have it in-house what's the opposite of in-house outhouse right so just keep that in mind when you when you're working with outsourced kinds of things so now we're going to get to the part that this is really about the problems conveniently broken down we're doing this a couple ways first we're going to talk about some pre-analyst problems these are things that happen before an event alert or something that an analyst needs to look into gets to first problem communication in silos the problem of uh getting all the data where you need to have it so so many people when they set up their logging or their

infrastructure they have i call them data cartels uh you know where they just kind of husband their data and they don't want to share it it's a lot of problem there feel free to join in and then you know it's always when you're hired as an as an analyst you're always told oh you know coming in you're gonna have all these uh training opportunities you're gonna be helped out along the way you can dive into whatever you want and then you start working the jobs and sometimes you were you know led down the wrong path and you're always told well your job is to monitor the network your job is to read respond to these emails and

sometimes you kind of feel like you're stuck in your your swim lane as our poor jimmy over here where jimmy's getting stuck in a swim lane where he really wants to progress but there he's always being stopped at every turn saying well you're you need to you do your job you've been hired to do and if you want to do anything else you need to learn that in your own time off premises it happens it happens in a lot of places right a lot of places that you come in the expectation is and from from the business side i understand this the expectation is you're getting paid to do a job so you come in and do

the job right comma but there's a lot of value in helping those people grow and there's a lot of value in helping your people grow so when you block them when an analyst is not able to learn how things work on an engineering side or how things work in the remediation side after they're done with something it's going to provide gaps in their knowledge that's going to keep them from being a better analyst in the long term and that becomes that big issue with communication and team breakdown in those silos yeah so i was going to say is a lot of the analysts i've worked with come from a different background they're either i.t guys never guys

so what they can provide and contribute is always valuable but they're always told hey just stay in your lane you know clear your alerts uh they're not allowed to interact with the cio team or it team and be able to provide feedback on how to get better data into our you know sim or sock wherever you're putting it and that's also really important to be able to get context around why things are firing in an alert and to get that you need to have that communication that open channel with the ops guys or or perhaps whoever's running that you know that system in finance they may be sit standing up something new if you don't understand that getting

those alerts really is kind of meaningless you're just checking the block at that point right yeah it's contextual understanding so in in the spirit of something like this and information sharing let's talk a little bit about ways that we think you can address that and my general thought process here and one of the things that i've done in the past anyway is i've had basically a team of analysts that i would use as external support and partner support that would work with my internal guys and in accordance with doing their own job i would task them out and this is stuff they wanted to do and learn they would get tasked out to work with

other teams and build formalized processes for information sharing and communication between those teams yeah one of the things we try to do uh in a lot of the stocks we work in is get the it guys to come sit in with us and see how we do what we do so they start to understand our purpose and our mission in there and then they're but they're buying in on this and then they're more apt to help you when you need a new data set you know sent in and it's also really important to make sure that you have a good security engineering team that can help bake into the processes to your configuration management processes and your uh

your crbs configuration review boards make sure that security is baked into that process so that when it when the system gets turned on for the first time the logs that you need are are pre-identified and you don't have to find out oh wait a minute there's a new syslog server out there what's it doing where where's it coming from you don't have to chase that down and that's where the communications having that open dialogue so often people are in their own silos where they just don't have that opportunity to to get out of it and this is where some of the junior analysts you can send them on expeditions go out there and you know

find out what's happening downrange you know you know give them some time to get out there and learn about the company or the organization that they're supporting and that way when those alerts fire they have some context around what's happening yeah exactly so when that's typically i like to push for typically i could always get stuck in those silos where you're stuck in that swim lane but i i hate it i always want to learn more and as i as i progress through my career within that company i always push for cross-training opportunities especially when you move into the more senior analysts within the sock shift lead stuff like that i like to bring in the engineers i like

to bring in the migrations teams and the pms so they can cross train with the analysts to see what they're doing especially when in the private sector when you have just the engineering team and you have the networking team and you have the sock sometimes there's like a breakdown of communication between the three where they don't really know what each of them are doing and having the opportunity as an analyst to go sit with the engineers to see like how the the sensors are configured or or the networking team see how everything is run and just having that that that let's say cornucopia of knowledge is important as an analyst so you can understand not only your network

but what everyone else is doing to support that network as well and tying that back to the to the to the prior discussion that's going to be different in an mssp versus a public sector versus a a government uh kind of environment you gotta there's different environments we'll be able to feed that information better than more and different information that you're gonna have so let's talk about information a little bit actually that sounds like a good segue that's a very good segue i love a good segue i really do i'm there for you man it's like practices or something so up next getting good data to an analyst this is a really big one and i believe

you want this is a big this is a big issue so a lot of times uh you go out you start building a sock you get all your data in but you have no context is where the data came from and then an analyst is looking at an alert that fires and they're like okay i need these other three pieces of data well sometimes it takes months or even years to get that data analyst if it's not configured from the beginning so one of the things you have to do is put value on your data when you get the buy-in from the it team let them know how important that data is to your analyst to be able to dig down

and get into whether it be netflow peak app full pcap log data from endpoints whatever it is they need you have to get that buy-in from your it team to be able to get that over to your site yeah you're going to want to definitely get that buy-in you're also you know if you can be there in the planning stages you know of the of whatever you know system that's being deployed whether it's a new web services or um server or something like that getting that buy-in early you know defining your requirements again i'm going to go back to the security engineering part of it making sure that all of those are defined and some of them are defined in

compliance standards anybody have to deal with this 853 just you know oh yeah yeah well that's all defined in there now granted that's not going to be everything you ever need but i mean it at least it's a ground rule that when they're designing and building these things you could say here's your compliance standards whether it's that or cramley's bliley or hipaa or whatever those things are predefined if you can get them to buy in on that however if you're mssp and you're working with some auto dealership out there probably credit cards maybe right you know credit um so that's that's a really important point right so one of these really important points that exist in

this space and one of the reasons why we kind of broke down different sectors of different types of socks too is because getting good data to an analyst and we'll probably cover this a little bit later but it involves knowing what your mission is yep you have to know what your mission is to know what data is good data yeah for example like we i don't know when the contract i was working on in the government somebody was insisting on importing all of syslog data out of all the windows machines they said we need it we need it we need it you need to figure out how to get it to the analysts we're like

but why they're like well we just need it so why is that going to be good data for the analysts well so and so said it should be done like you know how much data that is going to be over a single hour of traffic on a uh like 50 000 host network that's why from the mssp side when you go in and this doesn't always happen it really doesn't when you go into a client one of the first questions that you ask that client is what is it that you care about what are you trying to protect and this is true if you go into any job what is it that matters to you if i can

take what it is that matters to you i can then define what data that i need to use to make sure that i get you the results that you need and we'll cover that a bit more later but it's kind of relevant here too anybody else have anything else on this one or should we go well we can talk a little bit so tools and technology are another big thing too yeah how many people are at places with old technology sims specifically by the way sims are a big deal every everybody's got a sim barely anybody sets them up or keeps them set up right yeah no no one maintains the filters stuff like that

yeah tuning one place maintenance uh one place i walked into had a 12 year old sem that hadn't been touched in five or six years and uh so when we took over we're trying to get some new filters built and then the company said it'd be six months to build a single filter i mean that's not efficient for anybody so yeah stuff or you have to reset uh reboot the sim every every other day because the database overflows yeah that's always good and those kinds of problems aren't that uncommon either unfortunately those are the kind of problems that there's very little that somebody at an analyst level can really do to fix yeah it's yeah so let's talk about the

analyst a little bit yeah lack of staffing and expertise i'm sure this is one that almost everybody feels in every side of computer security we lack staffing we lack expertise that's a big thing has anybody broken the code on how to recruit junior analysts please let us know yeah so that would be great getting getting them in at the right you know yeah it's money it's money but then then the money you pay your analyst is going to come out of some other bucket from somewhere else in the commercial world and then there's the the the drive of that junior analyst do they really they just want to sit around collect the paycheck or they actually want to learn

right and so finding those folks so um you know i forgot where i was going with this it's the gray hair it filters everything it happens well and yes absolutely

they can't get a job because they don't have experience yeah it's a big problem and everybody wants to buy they want cheap but they don't want zero experience yeah so i think this is a mindset in the employers that have to change that's that's that's a big thing a lot of time we do see that a lot but as a more senior analyst you come in you have the five six ten years experience you don't have a problem getting a job but that's one of the things that we would like i like to push like within the hiring managers they we always need people there's always a seat that needs to be filled and they don't they don't you know they

don't have experience but most of the time i know personally when i interview somebody i don't care how much experience you have it's about how you think and what your drive is to learn and sometimes you have the people that can fool you but sometimes you meet people that are truly interested in learning and that's the biggest thing i run into is you know you have something right out of college they might come in as an intern or they might come in off of a recruiter and just learning to you know talking with them having that communication with them so you can figure out their mindset what they want to do and how they think is is a key thing is

i don't go in knowing everything i think we still have a little bit of old market method mentality right there's having a job to a lot of bosses is a privilege yeah you're in a good position because i employ you and you should be thankful for that but guess what we're in dc and it's cyber security there's way more jobs than there are people to fill them so that mentality absolutely has to change because it's not hard to go somewhere else all the way in the back in the green people

so so the question is is how involved before you hire someone how involved do you need to be with the hiring manager and setting the requirements is that that's the question so i'm going to say that you need to be intimately involved you you need to understand what the you know what the requirements of the job are before you go start casting the net and then you need to be able to spell them out for the hiring manager and have them go out there and pre-screen these things because there's nothing worse having hired people than sitting there and interviewing people who are obviously not qualified for the job and you don't understand how they got there or even

spending time with a resume you know the hiring manager is looking for the word analyst and suddenly you get some business analyst who submitted a resume well that's not the kind of analyst i'm looking for and then the other you gotta you really gotta educate them the other half of the problem though really isn't just the job description it's that more often than not at least where i've been they don't even define the levels of the analyst for the people doing the job yeah nobody knows the difference between the junior the senior and i'm not saying that you should put up walls that's the communications and silos thing don't wall people off but what i've done is you define a

system where junior analyst this is my clear-cut expectations you are welcome to do more you are welcome to do the intermediate analyst expectations and when you check off all those marks well it's really time to look at about getting you a promotion whether or not i can do that not always my choice but i'm looking to move people from one to another to another i love hiring juniors but until you define internally what your roles mean you can't really hire externally for it and you teach your your recruiters you i i've actually talked to recruiters who cold called me for a business analyst position because there's a word endless in my resume i've actually talked to recruiters and

taught them what a security analyst is and what to look for for a position that they're trying to hire within the security world and that's quite key you have another one of the question that i a lot of other people have been working on and this has been working on the tests that i'm working on is a creation of a way of identifying the words that you need to have in those resumes special publication 800-181 800-181 special publication repeating it for the folks uh the recording it's finding a way to quantify the skills necessary for analysts that are already in writing i'm familiar with 181 and that's in this publication if you've not seen it it's really neat it breaks

it down um the key performance indicators uh well ksa's kills kills skills something something something knowledge yeah there you go ksas it's all broken down if you've not seen that in this publication the other thing you can go into is i've had dealings with is the department of labor wage surveys it will tell you what a particular job is worth in every market which is kind of neat if you've not seen that at one point in my career in the civilian sector i was hiring security guards and what you pay a security guard in miami is different than what you pay in los angeles well i'm sure it's the same with security analysts department of labor wage

surveys yeah and then that will help your recruiters to quantify and and cull out folks you don't want to hire a 200k a year person for an entry-level job and that can you know that's part of that so if you guys don't mind holding for just a few minutes too we do have to get through quite a bit more although i love the questions i really do i think i just want to make sure we're good on time because we still got to talk a little bit about expertise and that becomes another big problem once you get in your junior analyst how do you get them expertise what do you do send them to besides was the comments we

promote a lot of this uh this community here i tell all the analysts uh get involved in the community get involved and unallocated get involved in b-sides go present because i tell you what if you want to present you learn a lot trying to get ready for that and if you got the money send them to some good courses obviously sans is great courses there's a bunch of online resources though that are free and you know i encourage analysts to spend some time if you have time at work give them a couple hours a week to go do something online that's free resource and i mentor them in that process let them go you know give them some

latitude within the scope of their job to experiment to explore as long as they don't break anything yeah go out there and they can start pulling on the thread as i call it you know there's you know they're working these binary alerts it happened or it didn't we'll let them go in there and start pulling on the thread and trying to figure out what exactly took place underneath the covers that will help develop the the skill and expertise inside and also keep them motivated yeah when you hire a new new junior analyst you know typically the senior you have the senior analysts who are designated to train and then there's senior analysts that are willing to train

and then there's a senior analyst that don't want anything to do with anybody talking to them they just want to do their job and get out for the day i think there's also the senior analyst that doesn't have time to train well that's the thing you don't have to have time but you do you know you might not have the time but you can always let them watch and if they have a question you don't you you're seeing as a senior analyst as a more senior person be a mentor and you see a junior coming out of college and they don't have the faintest clue what you're doing but they're interested that's the biggest thing you have the

juniors that want to just come in and click that button every day and go home with a paycheck sometimes you have junior analysts come in they they have that drive you can definitely pick up on it during the interview and within the first couple weeks they they're wide-eyed they're really scared but they don't know what they're doing but you need to reach out to them as a senior but like hey you know what are you doing give them the training that they need you know give them some packets let them work on some stuff let them shoulder surf you and then you send them off you answer their questions but most of the time some juniors come

in they're terrified to talk to the seniors and sometimes even the mid-level because we have our headphones on noise cancer and we're listening to our music we're just head down working so i'm gonna segue off one of those points but i am gonna say to his earlier point because i do hire people occasionally i look for people who do things like b-sides or hacker spaces or anything that they do in their own time because that to me shows me that they have a certain interest and a motivation staff motivation boom enthusiasm is contagious and so this kind of segues in if you are enthusiastic about your job it's very easy to have the juniors around you

um to be enthusiastic about their job um and it's it's really important that for the seniors in the room not to lose that sense of wonder and awe at what you're doing every day um keep up that motivation keep coming to training keep get keep asking questions if you're only there answering questions if you're going to work forward every day that will be contagious and that will emanate down if you go to work and you get up every morning and you go ugh i got to go to work then you're in the wrong line of work bottom line if you're not there if you're not enthusiastic then that will just permeate throughout a team yeah i get up every morning i love my

job because every day is different right so i always look for something in that day that i can take away and uh it keeps me motivated and i know that helps like i said it's contagious if you're up and you're right eye bushy tail ready to go every morning everybody around you will be the same it's very contagious but if you come in you know moaning and growing yep everybody else will too and and that works up as well so if you're working for bosses who are not enthusiastic have enthusiasm show that initiative either then you're in a poisonous environment you should move or you can encourage that enthusiasm just kind of asking questions uh in ways that get

people motivated to help you find those answers yeah pretty much just just take care of your you know as a senior you know just have the enthusiasm to teach to to learn stuff like that it's important to know that motivation is different for every single person every one of you in this room i'm sure has something different that motivates you in some way a key is when you're interviewing look for managers that have some sense of empathy that will actually figure out what it is that motivates you so that you can figure out how they will try to line that up with what they need from you because some people are going to be motivated by

learning more and intelligence some people are going to be motivated motivated by having impact some people want to be recognized people different things move different people and that's important to know as an analyst too the guy sitting next to you doing the same job may have a completely different set of motivators for them and if you can understand those you can much easily you can it's much easier to explain why people are the way they are and how they work wow that's a great point sean thank you see what i did there i want to uh i wanna flip this around a little bit too if you're out looking for a job as an analyst this is these are very important

things to ask for your whoever's uh interviewing you like how long have people stay with the company what do you do to help motivate people to stay is there a package of training is there a package for traveling out and going to these conferences some companies pay for people to come conferences that's awesome if you're leaving if you're going to a company you can always ask so who am i replacing and why did they leave oh that's because they thought i was a toxic boss well i won't tell you that how many how many times does that happen or they talk about how you know if you know the previous person left because they were overwhelmed they had worked

too many tickets or they too much was expected out of them burnout yeah burnout that's a big part of this motivation thing i mean there was just a study that and i you sent it to me and i can't remember the name of it for the life of me now i called bob but there was just a study that came out talking about general analyst burnout i've been in shops where analysts would work anywhere from one to five or you know uh one ticket every one to five minutes like that's an incredible burnout rate right like that's a lot of work and not a lot of time to actually dig in and do some analysis

and burnout i think is one of the biggest things we have and if i remember correctly that study stated or another one that i read that a general analyst when they queried said that they're comfortable doing seven to eight alarms a day so if you're going for an analyst job here's a here's a fun question to ask during the interview how many alarms am i going to be expected to work a day what's the average day look like from an alarm count perspective am i going to have time to dig or am i just opening and closing tickets yep anybody have anything else to talk about for motivation questions on that anybody anybody okay um the way that we fix all the unhappy

people is to fire them and then we'll only have happy people that was it yeah that was on the slide so what are we talking about so now we go into post analysts metrics misunderstandings this is a big one for everybody i'm sure right because everybody has to find a way especially when you get into the financial side in the business side there's got to be a way to justify your people how do you measure success yeah how do you measure your yeah so uh i was asked by a ceo one time how many bad guys did you guys catch this year with your stock right uh the answer was zero and he didn't like that answer he's like

well then you have enough staff uh actually if i had more staff i might have caught the one or two bad guys i didn't catch that i know were there right there's it's uh it's numbers they always want to know how many tickets you're clearing how many alarms you're clearing um and that's a big deal yeah it's always about numbers or tickets you're clearing it's it's not it's like the graph says you know jimmy closed the most tickets he must be the best but do you have yeah poor brian he's just not so good and you have erica and steve they're all right you know typically in the private sector you always have to have the justification for

the cost how many tickets did you clear how many tickets did you open and escalate how many tickets went out to the customer well so another fun interview question when you're interviewing for a job how do you run metrics on analysts yeah that's a really good thing to know when you're going in because that's going to tell you is it a shop where the person who closes the most tickets gets the best don't get me wrong i've been an analyst i've gained the hell out of that system i've done less work than everybody and looked like i did more and probably everybody at some point has done that sometime at night you're just sitting there working the overnight

shift and you're just like i know all these 400 alerts are the same uh close okay go back to youtube or something along those lines so so my approach at metrics is a little different maybe it's because i'm an analyst metrics are like an alert they are an investigation springboard to ask additional questions so in this instance brian closed the lease tickets did brian escalate more tickets are brian's tickets higher quality do i need to talk to brian maybe and see if something's going on what can i get and it's way more than just ticket closures that when i run a metric i run ticket closures i run escalations i run percentages i run basis on shift i pull all the data look

at the data like an analyst and then know what questions to ask next time from time from open until closed too is another big thing is brian could have the least amount of tickets but he only worked three tickets a day but it took him three hours to open that ticket and then come to a status change at least but jimmy's work opening a ticket in his 36 seconds to status change it could be three or four min like hours but you know brian's the least amount of tickets open so obviously he's the worst but he's actually digging in and looking into things always understand what kind of metrics you have when you're going into a new shop though

understand what they're going to value you on if there's a minimum take account a day and stuff like that yep question so what percentage of uh metrics do you use to identify flaws that are built into the system as an example a new program comes on and it's now doing errors you actually have a netflix for that or an older system is packed you know that also then causes problems the security is usually the first person's call management how do you manage that so the question is what percentage and how do you measure system metrics new systems coming online old systems getting patched rather than people metrics and you can do that through general alert metrics which is one way

if those systems are actually forwarding and alerts and you can use that data to let you know what you're looking for from a tuning perspective or from a correction perspective anybody else have thoughts here no no i'm good how much do false positives figure into the uh the analysts you know grade basically right oh that that sounds like well you have to identify them as false positives first right and then then that would imply some level of senior ad more senior analysts going through and grading the alerts that came in potentially afterwards or perhaps a a junior analyst nominating something to be a false positive that could also be a a metric that you could apply

for the junior analysts how many false positives did they recommend for tuning um and that could be something that they graded on as well that is a metric that i pulled there's what that is a metric that i pulled at places how many tuning requests we submitted right and what platforms those were for right so what he said so to go into the last part of this metrics ultimately come down to how an organization demonstrates roi metrics become the numbers that define the numbers which are the money and the money is what's important so how do we demonstrate roi this is quite possibly my favorite of the funny slides by the way so boy demonstrating roi that's a big

deal that's going to be probably less of a big deal in government but mssp and private sector big one big deal they're going to do that from a metric perspective so what can an analyst do to i mean i'll i'll just be straight about it game the system understand their metrics understand what they care about and you can totally game the system you might disagree with it and if you disagree with it it might be time to go it might be time to find a better system i wouldn't recommend gaming a system perhaps they could also recommend improvements to the system um but going in there and finding a return on investment uh that that is quite the challenge

dustin yeah so um in the government space they always want to buy the next newest biggest baddest blinky box right and they want to spend millions of dollars on this blinky box and then they want to hire two people to watch the blinky box and i'm you know i go in i think you're going to need 10 or 12 analysts they hire two so what did you just do you spent all this money on a blinky box that tells you all these cool things you have nobody to look at it right so roi for me what's important is uh scoping what you're gonna do getting the right people and i recommend a lot of times using

commodity hardware and building open source you know cheaper products but you have people that are invested into it because they're the ones building uh your shock they're the ones building all these data points for you building the analytics and that's much more valuable in a blinky box or they or they say in my opinion all the vendors that sell blinking boxes i'm sorry or they or they or they spend half the money they they they buy just enough blinky boxes to just barely cover the network right and don't expect as a side note we recommend all the blinky boxes of people who sponsor this show absolutely yes by all the sponsors well there's a place for them right yeah

there's a place for them um the problem is is when you buy a two million dollar blinky box uh you're going to need another million dollars in uh to the company for your tail right well this goes back to earlier like way earlier we talked about defining your mission right you can't demonstrate your return on investment unless you define your mission so if you haven't been given a clear mission that's something i encourage people to push for what is my true goal here what is it that you expect of me is it to find bad guys is it to close tickets whatever the case may be however brutal that is define your mission and then you can demonstrate your return

on investment by whether or not you're meeting that mission if uh if the organization is buying a two million dollar blinky box without talking to the sock about what it needs there's your pro here's your problem right there it definitely happens you need to you you know if they're just pushing security appliances down uh that's the untenables but more more technology oh my god the ais yeah well i got all the ais yeah sure no but i mean that's but that goes back to the the going back to the communication if you don't understand what the organization needs you don't understand what drives their decision process you can't influence it and if you're in a position if you're running a sock or

you're working in a sock and you're being presented with blinky boxes that you have to monitor and you weren't part of the decision process then something is fundamentally broken there and you really need to be and then that's an opportunity for you then to go quantify what your awareness is or your your perception of what's going on to management who because you have such a great environment because everybody's enthusiastic right i'm tying it all together here all right so you know then you have this good communication back and forth and get in front of it you know if you hear somebody because generally they don't just go out and buy a blinky box say you hear that that these things are

potentially coming down the pike have that conversation with your management if your management doesn't understand why you need or don't need a new blinky box then you know then it's up to you to educate them on why and then if you don't know what the blinky box does push to get trained on it so you don't just get in some training so that you have that training and the certification and retention yes a little bit of motivation tying that back in there look at that how that works yeah this kind of all full circles really and while we broke this down what we're really demonstrating is that every one of these this is a very secular thing

like and everybody's got to be on the level engineers have to be on level with analysts and the people who lead the analysts and lead the business need to be on level with them too everybody's got to know what's going on and it comes back to the first thing we talked about communication yes everybody's got to be able to have that communication how many people in here think they're good communicators yeah that's not a lot of hands it's not a lot of hands how many people in here has this will this directly affect in some way by the way like do soccer analysts work that's a couple more hands than last year i think so

we wanted to do this kind of from a community perspective and it's b-sides so we actually did submit we had so a few questions submitted that we can go over but we're actually doing way better on time than i thought we were going to do we can talk slower so no no way better time means more time for questions which means we can cover what's important to you guys individually so absolutely my favorite scene one of our online questions was do we think that there's more rabbit holes than there are analysts that was one of the online questions yes and that was not rehearsed no it really wasn't yeah there's absolutely i mean and that goes back towards the

depth of level of an investigation you want to pursue you know do you just want to close a ticket or do you want to figure out why the underlying pro thing occurred and you know that goes to the enthusiasm and the curiosity of the individual who's charged with identifying and triaging these things yeah we have the awesome capability in our systems we let our even our junior analysts go all the way down to the root cause right i want them to go all the way down and then the senior guys mentor and help them get to the right places but there's definitely a ton of rabbit holes and sometimes you got to reach over and

pull your buddy out because they're digging too deep yeah oh yeah they go start reading a blog they're like oh we're gonna find this and all of a sudden they find like an indicator somewhere that in the traffic they're like oh this could be we've been popped no that's you found the uh an a b x it was an av yeah so you're fine so just side note signatures fire on antivirus yeah you say so the other one and it was further down the the rabbit theme and was it was peter cottontail i don't know i don't know chicken little chicken little there we go do we think there's a chicken little between analysts and like sea levels technical people and

sea levels

oh yeah and that and that happens a lot when you have a very enthusiastic junior analyst um they will definitely see the sky falling and it's up to us as the more senior folks to to help mentor and guide them in the right direction uh you know just constantly constantly oh my gosh look at this it's happening um and it can be a great distraction for the entire team because you not only need to walk them back from the edge but explain to them why they're standing too close to the edge for something that's really not really even an edge especially in the government because you could have a gs listening to the npr in the morning he hears an article

and all of a sudden he walks into the office oh my word we're going to be hacked by board we've been called in after hours for something that came out on a twitter feed uh the whole team got called in we have to patch this right now we're vulnerable to all these things and we go and look and we're absolutely not but it happens but that's why you need to watch those twitter feeds and be in front of those things i've had a couple of times where something pops on twitter or you know some new report is put out about some new vulnerability and we get ahead of it before the boss comes in and you can be you know you can start

you know pre-loading those answers i i've learned that you know whoever gets to the boss first wins yes so if you can get to the boss hey you're going to see this news feed coming out your boss may ask about it here's what our status is you'll be appreciated more and and it makes your boss look good because they're in front of the issue that that the ceo might be pushing down and asking about yeah we also get some external alerts so some some especially in government sector there's other people monitoring stuff you're doing other places um an rco uh one time got an external alert and he thought the world was on fire but

once we looked at our data we saw it was just a false positive well that is something we didn't really discuss about as an analyst stay on top of data stay on top of the news feed stay on top of the blog post all the twitter just stay ahead never stop learning that's something that you need to do is never stop staying ahead of like the curve but there's so much out there it's hard it's very hard but you got to try don't ever try to become a master at one thing just stay ahead of the curve so now's probably a good time to open it up for questions and if anybody would be comfortable

actually walking up and speaking into the mic so you can get it on the recording that would be the best if not we can repeat that we can repeat this we can repeat we got a question in the back godless

so how fudgible is and i'm going to hit the first one first and then we'll do the other ones because i'm going to forget them how fudgeable is a sock analyst from one sock to another and honestly that's completely up to you it depends it really does because socks are so different everyone is so different i've been in intense hunt socks i've been in you know two inches deep socks so i guess the answer is it's not very fungible unless you can find two very similar socks that would be my takeaway i mean you it depends on your your your drive to learn new new uh go into a place that's completely different you have the skill set deep down but you

make the move over to a different sock that has your requirements at one sock was oh i just need to respond to emails then you move into another sock where you are from start to finish the person handling that event it doesn't matter what level you are so here's the better question do you really want to move to another sock and do the same thing or do you want to move to another sock and have do something better or more or something that you haven't done before so um another part of your question i believe was onboarding how long so uh one environment i work in there's about six months soak where you're just learning the

environment and you're gonna be doing things you're gonna be clearing alerts you're gonna do things but we we average about six months before you actually grasp the network and how it works so that's pretty crazy we also though uh where i am we have a two or three day kind of getting to know you training protocol where the new sock analysts come in and they they learn a little they get the network diagram they get you know where sensors are things like that and we train them on that and then there's actually i'm working with some junior military now there's a certification process where they get to sit before a board and kind of recite some of the things

that you know make sure that they're they're eligible to work on the watch floor solo and what was the middle question

[Music] best practices for dealing with shift work in a 24x7 sock uh don't make it your graveyard for the people you don't like yeah all right so because that that's it needs to be rotating shifts can you know there's it's a two-edged sword um you have people who like working the night shift um and if they're effective in that position great let them do it uh if they're not then you you're gonna need to find other places for them uh but you know i don't know working a rotating shift in a sock can definitely lead to burnout pretty quickly because you know you have somebody that works what four weeks on days the next thing

you know you work four weeks on nights your body never has the time to adjust and then if it's on a rotating schedule like that you're just constantly in flux and as an analyst it definitely wears wears you out and you just you start to lose that motivation because next you know like oh well then you lose a day and then you're trying to recoup the you work overnights during the week and then the weekends you're waking up at 9am to go out with your friends or with your wife or whatever so my thing goes back to communication so one of the one of the hazards of the off shifts and you and i at least have worked several

off shifts and we know that one of the big hazards of working that off shift is you feel like you're stuck on an island and alone yeah so the sock that i was just running by the way i'm employ unemployed until tomorrow so that's fun the sock that i was just running was 27 people five shifts 24 by seven and my big thing was i was on at least one turnover every week and at least online across every shift a couple times a week so if anybody needed something i was available and they could ask questions and they had a line to their management staff to make sure that they knew what was going on instead of hearing everything

secondhand and feeling lost because i've been there that's the biggest that's probably one of the hardest things when you work on an evening shift the bosses leave at five o'clock or earlier and then you're there until 11 o'clock at night or you work an overnight shift you feel like you're just sitting there and you have no management you have no one to talk to and all you do is you get emails down oh and the from upper management where you just don't feel like you have any way out and then if then it comes time to changing that shift you really want to move down to a daylight shift so you can feel like a normal human

being again and um you just you the only the only interaction you have with your actual management is if you want to stay after especially if you work overnight you want to stay after for four hours waiting for them to come in at 9am when you get off at six so more questions i know you do and i know you do anybody else i support your questions what do you got small medium what size business 50ish person business and the question is where do you need a sock as opposed to skilled it professionals and that's a really good question actually and i would probably say like a 50 person company right you're probably looking at a nine to

five fairly skilled security team two three people maybe getting alerts at night like you're gonna be up if something happens and you're gonna have that stuff forwarding to you but i mean you're looking at a two to three person stop how big is the it department two to three people here we go i see a perfect opportunity for crossover because if it's a commercial entity you're gonna have a hard time justifying additional headcount to deal with just security stuff when it's all like is the business i t related you could say that okay so so if it was like a car dealership or something you'd be hard-pressed to deal with to you know to justify that additional

cost to the management if you're an i.t group it's not hard to sell what the threats are and the vulnerabilities are so um if it is just internally that's where mssps make their bread and butter right there they can you can just become part of the status quo of an mssp and you just pay that yearly fee and then you just as the it people they just received the email or the phone call from the mssp saying hey you have this going on fix it just so you know that's the general target of an mssp yep so more next question don't be shy now we're fairly friendly four m we got four minutes until lunch starts but we can go as long

as we wanna i mean joe you can you you uh i'll take another one from you i know you got em every one of them how many red teams have you all of the above [Laughter] that they let us know that they're going to come in and redeem the network so it's just like oh watch out for the red team this week it tends to be hit or miss and i'll be honest a lot of my analysts will be like oh that's a pen testing team i'm just not gonna alert that because they don't care yeah they let us know that the pen the pen testers are be running you know activities on the network

and we're supposed to catch them okay the tools saw this happen it's got the it's got the watermark of the other the red team i'll write up one send it up the chain and then they're like okay cool you caught them and most of the time it's most time the red team's doing things just you know test vulnerabilities and network to help out with you know protecting the network but as an analyst it's just an annoyance sometimes because like really guys really i mean you have to be able to pay is it our job to catch the red team or is it our job to catch actual attackers yeah well if you catch a red team you

test your tools a little bit but yeah we've we've been on a couple tests where we didn't know they were there um they fired up a scanner or something and all of a sudden my dashboard blew up and i said what's going on yeah you know and oh there's guys in their office they're testing you right now good so it works great now can you tell them to stop please good

given that average detect time is measured in months and not minutes or hours what do you gain with the 24x7 operations i mean i think it depends on what you're protecting right there's some uh socks that run that protect really important stuff and then there's socks that run that try to make sure your you know pii stuff's not getting leaked out or something yeah data has a lot to do with it a lot of it is the pii i worked in the credit card business for a while and they watched carefully very carefully you know things that look like credit card numbers crossing the wire where they weren't supposed to be and you work for private sector

companies who have very sought after you know prototypes information you have to have people there making sure that there's not a an actor or a group out there trying to steal that data from you to you know essentially get get to the market first and having people monitoring for suspicious activity especially large sums of data leaving the network big one is is a huge thing because if you don't have somebody sitting there on a sunday morning at six a.m you know somebody across the ocean could be in the network actively exfiltrating data and you you it might not alert because they've found a way around your alerting tools but you can still see large amounts of

data leaving the network if you're monitoring you or that you also gain protection for low hanging fruit and i can tell you from consulting and probably about a hundred different cases most of the breaches that i went in on were low-hanging fruit it was email almost always email oh yeah or credential theft it's always something like that so the 24x7 gives you early access and early visibility into the things that will hurt you but not destroy you necessarily because you can watch uh emote is a really good example and i know jessica was talking about it in the keynote the spreader module i've watched that spread through a network in two hours and take out

160 out of 250 hosts before so if you can catch that early you can stop it early and not end up having downtime for four or five days well prime example as a junior analyst working in mssp sitting there it was a saturday afternoon headline news is on the tv and all of a sudden we see something about like a here you have virus come across the network and next thing you know you look at the you're looking at the sim and all of a sudden you just start seeing red alerts just start popping up and it's just like one customer to the next to the next to the next and then you're like it's all email and

it's all here you have wow gotta get some people on the corner on this one and it was a sun it was a saturday afternoon so there is value to the 24 7 stock which is that was also a really good question thank you emma do you have a question oh she's ready for lunch emma says it's one o'clock it's time for us so then let me let me pose the question is everyone ready for lunch yes because i'm sure you are if you have any other questions you're more than welcome to come talk to us we're gonna hang if you have questions i hope everyone enjoys the rest of their time at b-sides and thank you all for coming out

thank you