The Security Operations Command Center of Cool - Lessons Learned Vol. 1

all right then ryan with that uh just special thanks to ryan uh you know our old uh our old greenville friend and that uh headed off but still is very much a uh greenville resident at heart and uh he volunteered to uh um speak today when he was the first person i went to when we had to make some changes to the schedule so uh special thanks to ryan thanks again for everybody for showing up and coming and uh without further ado i'll turn it over to mr bazel all right thank you mike it's good to see everyone today and it's uh good to be back talking to my fellow greenville colleagues uh to we're going to cover today

something that i've learned in my time doing security operations in professional services it's going to cover basically two organizations that i've had the opportunity to work with while doing some consulting work and um hopefully obviously the goal is is some of the lessons i've learned maybe if you're in security operations you have the opportunity to apply some of those if you're not in security operations maybe it's it piques your interest to to work in that that field and security and i know many of you are in college um or working on uh getting into the security field and this may be just additional information for you may peak your interest as well to explore this field further so

just quickly a little bit about me and my personal purpose i'm a husband a father of two my oldest turns two tomorrow and then i have a four month old as well i am a consultant i do consider myself a relationship builder as many of you i know here on this call have good relationships with and know very well i am a christian a lifelong learner i do love working with non-profits i am as mike said a b-sides greenville organizer and i enjoy to travel so i went through this exercise at a professional services firm and i just i want to share it with you all because um and it also mirrors a book i've been

reading stephen covey's seven habits of highly effective people not really related to talk on obviously security operations but maybe something just to encourage you with is to develop a purpose statement in life and what you do and some actions that you're guided by so my personal purpose here is to find life's meaning so that boldly we reveal our true self and when at my best i bring my purpose to life guided by a set of actions i ask questions i encourage others and i serve others and by living my purpose every day i actively build a world i imagine a world in which people seek out new opportunities that challenge their potential people feel validated in their decision

making after receiving counsel and being encouraged and people consider thoughtfully on what they want and are empowered to pursue their desires that transitions to our purpose today to where hopefully we accomplish three things we explore lessons learned from security operations we share about what life was like working in security operations and then like i said we encourage people maybe who are early in their career or who are looking for a career change maybe to try out stuck ops so the first lessons learned involves security operations operating model so like i said i worked at i worked in two organizations and i'll describe these organizations to you first they're both life sciences organizations slash pharmaceuticals both are multinational global

one is based uh headquartered actually in the apac region uh the other is headquartered here in the us actually not too far from where i am here in chicago and both you know extremely large revenue-wise large security teams both though have different operating models which was interesting to to be a part of the first organization their operating model and i guess i should describe what i mean by operating model it would be probably obvious as i went through but it's how they're structured um team wise so the first organization pharmaceutical company they had a u.s team lead and then they actually used an outsourced sock with all of the analysts sitting there in romania and they used a leading sock mssp

provider operated 24x7 the second organization i worked with they actually did things a little differently they had a us-based team lead but then they had in-house folks in global delivery centers bathe both in central mexico and in india in each each location had four to eight analysts and they covered um a 24 hour period and and this shared service model global delivery center model was really helpful to them because not only were there stock analysts and ir analysts there but they had network folks they had forensics folks general i.t practitioners as well so when we needed something we were able to quickly go to those individual teams and the people in those locations from an ir

security operations perspective knew who we needed to talk to and had pre-coveted relationships with them and you know would sit in the same office the second organization i worked with they were going 24 by 7 but one of the issues we did run into was kind of a lessons learned was staffing is really difficult and many of you know this here in the u.s and it is extremely difficult in in india and mexico as well it's probably not a surprise to that we did have some some turnover there we actually had to move back to 24x5 and make the the adjustments needed there to to accommodate so lessons learned quick highlights is that very of

varying operating models can work whether it's all in house or whether it's outsource i did notice that the bond was definitely stronger even between me and the team when everybody was in-house the teams were obviously stronger because they knew one another but like i mentioned some of the challenges were you know recruiting was hard the quote going rate for finding talent in both mexico and india has gone up significantly um so and this organization did face some legacy hr challenges with getting equitable pay equitable pay based on current data and another interesting thing many of you may not be aware of is that in india when you're trying to hire someone there's actually a 90-day notice period

and this led to you know this organization may be making an offer to an individual them then having to wait 90 days or they may have already been within a 90-day window and they were simply shopping around now that they had an offer in hand trying to find a better offer so the next area lessons learned bucket is the area of process and the importance of having process and and i did this mostly at the second organization that i was with where my responsibility when i went in was to develop processes that were not in place and so we focused really on four buckets or so of documentation that the team needed that was not currently in place

so they were triage playbooks investigation playbooks threat hunt playbooks and then we also had operations type playbooks so the threat or the triage playbooks these were more focused on how an analyst would action an alert based on a tool and weeding out noise one of the things that i thought was was new to me was using a using onenote or some other tool where we had kept a list of known behaviors in the environment so this was certain applications that may run that may gener generate alerts in the edr it may be specific processes and processed trees that would flag alerts in the edr solution so we wanted to document those so we could easily you know analysts could

review the known behaviors at the beginning of the day see an alert and realize oh that's something we don't need to worry about we can set that as ignored we also had a known malicious behavior one note notebook that was based on minor attack techniques that the analysts would also review against investigation playbooks these focused on kind of deeper analysis and we had originally set out to match each category outlined in the verizon dbir which ended up being like 11 or 12 investigation playbooks that were in scope to start and as we started writing them we realized for this organization we could really bucket those into three that was malware ddos and other and we tried to go in and answer

key questions and those were is this was malware focused is this ransomware credential theft has activity been automatically blocked by a tool is the activity new is the activity spreading or localized is the activity malicious what's the source and is the threat action internal or external and so we would help guide the analysts through these steps and these questions trying as as hard as we could not to be restrictive and you know perscriptive so that analysts would still be able to move and and have gut reactions and make their own judgments based on what they were seeing that was a very delicate and fine line to try and walk operations wise these may be better

titled as job aids depending on your terminology and how you do it so these were based on specific tool actions and tasks tasks that you may accomplish for the various security tools metric reporting we had a daily handoff that we did that you know we would document metrics in there and what to talk about we documented threat hunts performed specifically in tools whether that was a proxy service the edr email and we had those and then one of the items that we actually developed much later um and it was the recommendation of a a colleague who joined us um and was hired on was more of like a cover book an ir sock cover book and this was kind of a

one-stop shop for a new hire and it contained things like an operations guide for the org chart what the tool set was and then the list of other documents that we had compiled at that point so that when new hire came in we could say here's this document and they had what they needed to go and start learning about the documentation set for the for the environment so lessons learned you really cannot have enough documentation i know and we try to keep this in perspective of a new hire someone who comes in and may not have the full understanding yes your your mid-level and your senior analysts are not going to pick up maybe the investigation playbook every

time but that new hire who's just coming in is maybe getting familiar with with process there's not enough documentation that they can have that would not help them so that was key we had to be very thorough again walking that fine line between being prescriptive and giving leeway for analysts to make gut reactions and then my colleague had these these three points i call them the his abcs when writing documentation make sure that there's accuracy brevity and clarity in in your writing and i i write that down and i keep that in mind now as i write documentation because nobody loves 100 page documents like you get with some reports and processes sometimes you can say a lot less in

seven or eight pages

all right the next kind of lessons learned bucket was reporting and keeping track of stuff again i'm going to focus on the second organization i worked with and some of the various items that we we kept track of and reported on and how we did it i think that may be helpful for some of you is one of them's major incidents and first of all i say if you don't have a ticketing tool you've got to find yourself a ticketing tool whether that's super big service now when i was in legal and my former boss maybe on the phone i don't know we used something called cherwell and it was just a place to keep stuff

and track stuff absolute must but then there was some things where we needed a little more flexibility and we turned to one note for that so for major incidents what we would do we would use a onenote and after triaging and realizing something was a little bit bigger than normal we started to use onenote to do a few things we would start to develop a timeline as far as what happened when it happened how it happened we would start setting up kind of decision trees decision log we would draft incident summaries in the event that we were going to the cso legal other it leadership so that we had you know six sentences or whatever it

may be to give them a high level of what was going on and we found one note to be really helpful for that we also did daily handoff reporting again working with a team in india and mexico and having them overlap we needed to be able to make sure that anything that say the india team worked overnight for the u.s hours of the mexico hours was handed off and any incidents that needed to be followed we used one note for that any threat hunts that the team did that may need further analysis and we would incorporate the daily handoff reporting used in onenote into into a call to make sure that anything was seamless there and we would have a daily call

with the the team and then we would have a handoff call between both teams to make sure that any escalations and such were taken care of um another thing for keeping track was a daily shift checklist so this was kind of supplemental to our operations guide on the daily handoff and this gave guidance for naming conventions test call procedures for a sock hotline checking any other relevant mailboxes a phishing mailbox a general security mailbox whatever it may be and then trying to give a priority order for what to review and then that kind of all wrapped up into a weekly status update where we gather highlights talk about hr people items development review any new threat intel that had

come in go over ir items of interest and then there would be itops related items as well so lessons learned much can happen over the course of the day many of you security know that and if you don't have someone working overnight a lot can happen between six o'clock and six am so just making sure that you have everything from the prior day documented is crucial and then let alone keeping track of things over a week over a month over a year and trying to track trends um one of the things that my peer colleague was very heavily on what in instilled in the team was looking for anomalous metrics and what i mean by that is

looking for what is rare and not necessarily rare just in general but what is rare in your environment so for instance we would do uh lol bin hunting living off the land binaries and we would look for executions there were certain lol bins that rarely if ever kicked off in the environment um and you know our edr tool we were able to go and search that across all lol bins over a 30-day period well we would search and we would have rules that would kick off if something flagged that over a 30-day period prior had not so that was considered rare in our environment so it may be something to to think about for your environment

and then i mean encourage your teams um you know i know from a leadership perspective we don't want to be super prescriptive and you know you must do this we need to give teams flexibility but encourage as much as you can for them to document as extensively as possible that's within reason and make updates to that documentation as much as is needed because they were often times we would find ourselves working an incident from maybe the edr platformer in servicenow and there may have been someone from the team who was off shift that day because they didn't work monday through friday they worked like monday they worked four fours if that's how you say that or four

ten hours or whatever um so if they were off a day but they had a ticket open and there was something that kind of had come up there were no notes we were kind of stuck so just encourage your teams to document our next bucket is transition to operations and this goes back to the first organization that i had the opportunity to work with and my colleague he called this tops so you take transition to operations you get tops and this was kind of our process for building out moving from one managed service provider to another managed service provider and so we would have a document and this managed service provider number one they managed all the tools

processes etc we were moving to another one so the document included things like making sure all architecture diagrams wiring diagrams ip and url information vendor information key processes such as health checks break fix config and change management patch and version management service requests user administration those key processes all of that was documented over here with organization mssp number one and moved to organization mssp number two um and this was was key to make sure that everything was was seamless because at one you know at a certain time organization one was going to go away completely and any historical knowledge they had poof was going to be gone um and so this was this was a multi-month

project um and just kind of how we did it was we obviously built our plan to transition um we had to know what work products we wanted to have and then as we kind of move stuff along with training and knowledge dumps and transfers we hit the point where we did a a passive parallel run where each organization was doing the same thing and there was some communication and dialogue between the two and then when we were ready to kind of sever the ties we moved into a stage we called hyper care where our focus was on the second organization picking up and going forward and then um the second organization going forward and and doing everything with very

minimal if any help from that first organization so some of you may be thinking oh i'm going to transition providers could be even just managed i.t services it doesn't necessarily have to be security related but having some type of process for transition is going to be key to ensure a smooth successful operation

my my former boss who i said maybe on the phone here i'm not sure he would always tell me this and he would he's former army and george patton is credited with it you fight like you train so one of the things key lessons is train your people and in the second organization we how we approached this was we actually had developed a skills matrix so we took and kind of identified 20 or so skills that we wanted our analysts to have and to develop to become better security analysts so what we did is we kind of mapped we created a mapping or a matrix and then a catalog basically of the of free courses

of resources of b-sides material from across the world and then we mapped that in videos and talks that had been done we mapped that to specific skills from incident handling network forensics memory forensics windows systems linux systems you know the list goes on that an analyst would need to kind of have understanding about and when working in the environment and so then we encouraged them we had paid stuff we had free stuff and then we had required courses that were outside of you know typically the general bucket of hr training that analysts should go through then we supplemented that with group training one of the challenges we faced there with group training as you can imagine spread across the u.s

mexico and india was trying to find a conducive time that all of the team members could do that so we were in three time zones i think at least three time zones two continents and it ended up being that our folks in mexico they had to start at 6am and our folks in india they would end at like midnight so we we and that was all they did that week the the other team members covered the shift so you got to have a plan in place for that and then let them focus on their training the life of the packet this is a lesson that i've learned actually at neither organization but from a colleague who used to run the

cisco sock and he termed it it was actually an internal initiative i worked on uh when i was in professional services and he turned with the life of the packet and so we he kind of broke down this framework that hopefully you know maybe gives you a different way of thinking of something but he broke it into operational tactical and then strategic and so in operational what you would do is you would map each data source you have to an outcome and then you would think what did you hope to gain from that so for example you would have um in general say net ops and you would think of like cisco stuff or you'd have your it ops

and you'd think of servers domain controllers and then other app servers whatever you think of your sock and you think of endpoints firewall logs network activity windows logs from logins who's accessing the network so you have this data and where it is and and how the packet a data packet flows because everything's in a packet then you move to tactical and tactical was really the time to detection and you have to have the right data source to to know what's going on then you would map that data source to having someone with that right skill set so you have palo alto well you want to have someone who has palo alto or network security experience

in the networking space to be able to help you understand the detection of that same thing with windows you want someone who's an expert or a professional in windows and then you'd map that to your outcome which is kind of the strategic option or this maybe the uh indicator that you're looking to get so going back to our net ops example netapp cisco data you would typically get from that that would help from a security perspective is i p address and poor itops the server domain controller you're going to get dns information you're going to get account info you may get asset subnet information and then from that sock standpoint you're going to get that endpoint

information which is going to include hostname potential hashes if you're using an edr tool uh any urls maybe and then that strategic phase is going to be that time to medication time to mitigation and this is where he had the the saying that you wanted to automate the known so automate known responses as much as you could and then you would investigate the unknown i'll say that again so you want to automate the known and investigate the unknown and then one of his final nuggets that i wanted to throw out here for you all was his his what are the vectors of threats and he had three and this simple if you think about it

you had internal to internal threats internal to external threats and then external to internal threats and all of those kind of weave together to understand that life of the packet

all right this is our last lesson here this is being introspective with data and one of the this is it was a key item learned at the second organization that we were maturing before i left and the lesson i learned here is to really dig deep into what you have so look into the data what types of alerts are you getting what hosts are receiving alerts what geographies are receiving alerts and what business units or people are receiving the most of the alerts and once you you know start mapping out this data and looking at it and if you're tracking it in a spreadsheet or service now or your other ticketing tool it's going to

make it a lot easier but what my my colleague encouraged us to do was to try and figure out the trends you know and this you know very popular for like email and business email compromise you know you're having a lot of folks target people in finance or target vips executives but from in this organization we had we had operations in china you know what about maybe these this chinese sites because they're multiple manufacturing sites you know was it different than say manufacturing in india or manufacturing in central america or south america um and so we we were not quite there yet to be able to do this well but it was something i wrote down

you know as i moved on to to keep with me going forward in other organizations is to really try and dig into that data and see what trends you can pull out this goes back to my comment earlier about trying to find what's rare in your environment what what ip addresses are uncommon what domain names are uncommon you know this organization they'd have millions of domain name resolutions a day you know but trying to find those ones and twos that maybe had lengths that were abnormally long and being able to do do analysis on that and see if those were trending over certain periods of time that was all key to us it came really

became really key when we had an incident like solar winds and trying to search back through data so final lessons learned here were to really document the types of alerts the locations document your thread intel you know being able to go back through if you're using like a tip or something great this organization um was was trying to mature that process as well and and feeding that in a lot of the thread intel is is very noisy as many of you know so trying to figure out what is obviously actionable is is difficult but when it's stuff happening in your environment they maybe see over time maybe a little more actionable than something you get from an isac

um challenges we faced here were finding rare occurrences in a large environment can be really hard like if you're looking for you know base64 encoded powershell because you think that you know very popular for kicking off a c2 connection and a downloader if you don't know what the basics before is and you decode it great um some tools are starting to do that some still don't but what we started to do was actually take the base64 string and document that and copy that out so as an analyst became familiar with what was normal something that you know first six eight characters of a base64 string you could tell this was different and they could pick up on that and then

they would know hey i can skip this this this oh this is different let's throw this into cyber chef or some other uh base64 decoder and see what we got here um and then you know invoke expression or iex are those used in your environment those are are very popular for adversaries looking for the term string and i learned that from the offensive security person there and one of the other ones i learned from an lol bin perspective is in run dll32 you you know command lines run the ll 32 happens all the time but what you're looking for to to maybe find or start with identifying malicious behavior is looking for something that says start

so if you find something that says start in a run dll32 command line it is actually starting something and if you want you want to find out what that something is and in our environment you know it was 50 000 hoes over the course of a day we may get one or two items that said start when we would have paired down prior to pairing it down a couple hundred thousand lines potentially after some filtering so a lot easier to investigate when you're being introspective and digging into that data than just kind of doing full bore scopes and not knowing what you're looking for i would say that was the other challenge is what is normal what is not

knowing what is bad was also a very difficult thing to kind of go through we were lucky at that point to have someone in offensive security who was able to help guide us through what he would do and what was maybe mapped out in mitre or other tools like vector for purple teaming that would kind of guide us to know what we needed to look for so with that um i know that's not you know a full full hour talk today and i told mike when he asked me on i think it was wednesday um i wouldn't have a full hour talk so i'm happy to answer any questions you may have um i do have a few minutes and

other than that thank you very much it's been a pleasure and i'll turn it back to mike thanks ryan anybody have any questions i know uh luke had one in the chat that i'd bring up but wanted to see if anybody else had any other questions anyone all right so luke asked so when considering equitable pay i'm curious how this works when you're hiring remote infosec professionals in different areas of the u.s or the world you pay them equivalent if they were local or pay what the skills work are worth in their home area so this organization um and i think other organizations probably like them they pay what is what is the going rate and their home

area so in india obviously currency is is very different um it is it's currently based on the the home area like i said it has become um you know for example like it's it's almost two or three two and a half times maybe from even just a couple years ago what i base maybe recommendations from what i gathered in my conversations with the the team lead um from his conversations with hr two at least two maybe two and a half times what someone in that area is a couple years ago would have been paid should be paid now so and it's taking this organization time to kind of get up to that and realize to find talent um and find the talent

they wanted they were not able to pay what they paid a few years ago sure i would definitely see the the need for for cyber security and that awareness only growing especially outside the united states where that just wasn't the case even a few years ago and i think that definitely is having an impact in those salaries as well which is great right for for all of us as cyber security professionals one of the questions i was going to ask is do you have any references that you use when developing the different run books and playbooks that you were creating for the sock teams so we based everything on mitre that was the the framework

that they wanted to use um really the coming from professional services we had reached out to others who had done run books for other organizations as kind of guides we took some bits and pieces from that and then really it was a lot of collaboration with the key stakeholders to understand what would work for their environment and what we needed to do there so it i mean there wasn't like formal standards so to say um a lot of it had been developed internally took bits and pieces from that a lot of maybe some stuff online sans has some stuff you may be able to to massage and finesse their stuff's usually a little more broad um

and then a lot of collaboration hours of collaboration with stakeholders sure which is good which is how it should be right yeah i mean at the end of the day in professional services our goal is to deliver what the client wants and what the client needs and kind of marry the two together and help them maybe see what they need um so we don't want to come in and say you must do it this way because that may not be the way that works for them sure yep that makes complete sense all right last call any other questions yes i did have a question can you all hear me yes dorian hey hey ryan so i had a quick question

so you mentioned that training was one one thing that you definitely would like to emphasize yeah um so regarding regarding that do you think that it should be up to whoever is doing the training or do you think there should be more of a generic standard for that training just to make sure you know they're the best results um so i'm not sure i understand the question sure completely okay let me clarify i'm sorry so i'm saying like um do you think that say for instance is just a company trying to train do you think that there should be do you think that they should put you know have their have their own idea of what

what their what their workers should do as far as training or do you think there should be a general um standard over the companies as far as how they should be training so i think it depends on the role you know we want to map training to a role and we want the that training to be applicable to the role so and it may depend on sector you know what specific training you do so i don't know if you can make the general assumption that you want to have standard training across all organizations i think it i think it's specific to the job specific to the sector specific to a role and then you know one of the things that i think

many of us in security know is you really you really can't just depend on probably your organization to train you right but budgets are tight there's stuff you probably are going to have to do outside whether it's i mean i leverage my local library all the time to pick up books and to learn stuff home labs um you know i'm gonna plug chris sanders applied network defense his trainings and his coursework out there is is top-notch and it's relatively inexpensive you know his his couple hundred dollars courses you know they'll compete with a sans course obviously it's going to be different but you're not going to pay 7 500 you'll pay a couple hundred bucks and

you'll get really good stuff scott and the chat i know i know scott scott's a big fan of black hills information security and they do pay what you can i think they just came out with another one not too long ago as well so you've really got to get out there um and i mean i'm going to plug our own b-sides we do some free stuff and we do very inexpensively paid stuff as well from top-notch instructors like michael holcomb so and others and get involved with groups greenville b-sides got isa um i know luke's on here i think i saw um maybe eric defcon 864 get involved with that or wherever you are i'll look for groups like that

sure okay sure thank you so much um thanks so much ryan yep it was a great question thanks dorian anybody else alrighty with that then thanks again for everybody for coming special thanks for ryan uh for not only presenting but definitely all the work he does in on on b sides and get to see the reaped awards of that hopefully in a couple months coming up