Implementing a Kick-Butt Training Program: BLUE TEAM GO!

next up first for the afternoon we have ryan chapman with implementing a kick-butt training program lutein go my good oh there we go hey folks how's it going today good everyone's good you're not too tired after lunch like laggin gonna fall asleep right right everyone's alright a good deal so my name is Ryan Chapman and we're here to talk about Simpson this five-week training course so they put together I was a trainer for almost like 6 years like a technical trainer you know corporate trainer and I do instant response now I've been doing it for about six years but enough about me you don't care about that so why baseline training we're gonna this is here's our agenda no one even

cares about the agendas either but there's two items in bold that are very important here one of them is the core tenants of the program I'm sorry I'm moving around a lot well bad I got to see the camera guys look come on Oh so the core tenants of the program and then the five different phases are basically the five different weeks and we're gonna talk about how like how they're designed why they were designed that way their module you can kind of move them around and they're flexible and blah blah blah blah blah so here's a little bit about me I absolutely love retro gaming anyone here like a retro gamer huh look at there's a picture of a

power dog I'm using a power gloves hack is that so I at one point I have a problem with collecting at one point I actually had five sick a Dreamcast five of them and I thought that was a bit Daffy so I sold two of them and now I only have three and that's just far more logical right I just still have three I don't know why but yeah I also also how do you know if someone does CrossFit they're a doctor or in this case they do Brazilian jiu-jitsu that's they'll tell you or they'll include a picture in all the damn presentations alright so there's me getting choked out I still get choked out pretty bad I'm actually not very

good but I love it so yeah know enough about me this is an 80s retro gaming themed conference right so I was like the power glove everyone I figured everyone was gonna play on that movie kung fury he watched kung fury right and so I had like a hacker man shirt that I was gonna wear and he wears this in the movie I was like wait a minute I know what I can do so I modified it and I basically just pulled it apart took all the stuff out and I put a little presenter remote in it and now I can just go oh great is that and I enjoyed that a little too much

so why baseline training first off anyone here supposed to be participation right back and forth anyone here throw your hands up if you are actively on our blue team now or you have a blue team at your place of employment okay how many times will blue team or come in and know exactly how to do the job that you want him or her to do it's extremely rare there's so many different frameworks you can follow there's so many different methodologies and ways to approach a blue team the defense side of things you can't just pull someone in opposite half the streets so to say educational institutions and certifications are typically rote memorization you pull these folks in and they're like I've

seen this content I've got like you know six different certifications and you're like okay that's fantastic how many bits are in a byte they're like what no I don't know there's been so many people who have come into our place with like graduate degrees with forensic analysis some graduate degrees and like the building block basics they just they just don't know them so that's part of why we designed this at my quote-unquote day job but I'm not here from day jobs let's just move on so the goal for this is that red teamers when you hire a red teamers typically they come with always these varied backgrounds right you might have one gal whose primary focus is like

just web application firewalls and she could pull the heck out of those things right and typically what red teams do is they hire a diverse set of people blue team's a lot of times as they try to do that but then they don't ensure that every time they're responding to an incident that the same steps are being taken to have kind of like a unified defense so that's what we're trying to kind of get over we're trying to fix that essentially so speaking of which we want to develop people who are Jack's of all trades Jack's and Jill's of all trades now some of you might be thinking oh we don't have the personnel for that

you know we have to we actually have to compartmentalize if you will well a problem with that is if you have just a deeper team and then you have a reversing team and then you have the tier 1 or tier 2 triage responders the problem you run into is you know something comes in it's a weaponized JavaScript file you know a PDF of JavaScript and they go oh I can't do that I have to wait for so-and-so to come in they work in three days and meanwhile that threat just lingers and just kind of hangs out and sits there in the network and that could be a bad thing so again day job slash what I really

like to see is a responding group where everyone's capable doesn't matter what comes in doesn't matter what that ticket or that email or whatever it was comes in able to handle it or someone on the shift is able to handle it and in the kind of work in unison so the goal is essentially not necessarily like tier 1 tier 2 tier 3 and capabilities matching that tiered group but rather anyone there can handle the situation a lot of times in security we have very few people in order you know like how many people here again with your blue teamers how many have more than 5 people I saw really that's all a couple hands alright

cool so a lot most folks don't they have one or two people sometimes you know so having a varied skill set is very important alright to start off you're gonna have to have some type of subject matter expert or a trainer to train the trainers so if anyone wants to start looking into this there's a program I forgot the name of it it used to be a STD now it's a TD it's a technical training organization they certified trainers and one of the things they talk about is a t3 session where you train the trainers so if you have sneeze like from reverse-engineering host-based forensics network forensics or what have you then those people would be the ones who

are training your trainers and the trainers will then be training the analysts or you know so on and so forth alright core tenants this is this is the really important stuff right here ok we want people to understand the how and why behind what they're doing too many times we have people who can basically like like a technician right I don't mean to bang on geek squad but just the name came up right they may know how to replace an FF are you a field replaceable unit but they had no idea how that actually fix the problem you know their program might say like I pop it into graphics card and they go ok and

they put a new one in and they're like problem solved right what was the problem we want our text and our analyst especially our responders to actually know why so we don't want silly monkeys hitting buttons and getting bananas all right we want intelligent monkeys wow look at that guy he's got an idea alright so this is what we want we want people to understand how to do things and they typically learned through experience all training who here listen to the keynote yesterday dance right Rachel play the first name right Rachel Toback she was talking about how she doesn't learn something else she actually does it hands-on does it and those of you who may be familiar with

Malcolm Gladwell and it's 10,000 hours concept unless I'm not doing insert response for 10,000 hours but the idea is the more you do it you're going to understand it a lot better so we want to foster a go here to find the answer mentality we have a lot of people who become dependent on others especially when they come into a tiered infrastructure or team if you do happen to have one and the tier ones are typically like well how do I do this you know where do I go to get the ant they don't ask where do I go to get the answer they ask what's the answer and they become reliant on that capability well we don't want that we

want them to know where to go to get that answer be it internal or be it external a good example of which RFC's this is not meant for you to read the text or I know it's all small don't worry about that but this is an RFC for HTTP the protocol so understand the protocol at a lower level at the very least you need to understand response codes right what's a get request what's a post request when you're initializing an initiating an SSL connection why you'd get a connect request what's the purpose of all that where the response codes that come back as opposed to answering a question when someone says what's at 3:03 and you go oh that's a

bell you get all happy near like oh you know no like go over here there's an entire list of them they're all there all right so we want them to give them the tools to answer their own questions going forward if you have your own knowledge management system you know a glorified wiki basically if you have a wiki whatever and you point them to the articles it's over there go find it over there and then they find the answers themselves hopefully going forward all right different types of learners come into play apparently this is now debated some people say this is bunk it worked for me for years so whatever there are visual cues that people will

give the trainer typically that should kind of show how they learn best kinesthetic learners they learn by doing and often when you ask them a question they will literally like look down and they're trying to visualize themselves performing that particular action or activity if you notice that in a training course or whoever is doing the training that you pick up off the idea here whoever is noticing that can kind of give them assignments or tasks that involved actually doing auditory learners will typically look to the sides and some in fact there's a guy hey art where are you hey I've got a guy back there that I work with and whatever he I tell him something if he really

wants to say what do you say to me he'll kind of go like this and look at me like okay rephrase it again he'll be like I got it he's a really auditory learner and I just kind of picked up that on that whenever just communicating going back and forth so you also have your visual learner anyone here look up when you're thinking a lot of people are visual learners and look up so depending on what type of learner the person is and you can identify just with our visual QS you give them tasks and assignments within the training sessions to kind of foster that you know someone who may be a kinesthetic learner or a visual

learner you know you start talking about like having them draw diagrams on paper or showing them diagrams and having them replicate them things of that nature everything should tie back to the real world so whatever ticketing system you have would however you manage your incidents whatever it is you want to bring those up as examples like let's just say you know say you have Splunk and you say we have 147 different alerts that are active right now and Splunk okay that's great how many of them usually fire or do the cop top ten they're like oh these six fire all the time okay use those use those as an examples go over two of them

each or something like that and make sure that why you're doing that excuse me you're also reviewing other people's work ticket review and shadowing are critical especially because they serve as really good jump off points for your training so you're going through a ticket and you're like hey we talked about DNS like look here's the actual query type right here in this pcap and they go well how come Bob about that and they use that as a jump-off point and start taking notes on what you covered with that particular individual so you can cover it with the rest works really really well shadowing is also indispensable if you have the personnel and capabilities and overlapping shifts

to be able to do that so the framework I come calling it a framework it's really cute I just came up with five different weeks of training okay you can do them all the one oh sorry you can do them all at one time you can do a couple weeks here and then a couple weeks there whatever if you have someone who's brand-new the reason I'm calling it flexible is if you do fall five weeks of training a row which includes host based forensics and includes Network forensics and includes worse engineering up like dotnet code and all that it's gonna be extremely overwhelming so I don't really recommend that all right here are the five weeks

here's the breakdown week one is intro setup and this sim I'm not checking my email by the way I'm just checking my time real quick that I never started cool I never started the timer moving on alright so week one a lot of companies you know HR will do like a half hour session and say hey the work company X you know we hired you because of whatever and thanks for being along and you get paid on these days right we're going to talk a little bit more detail the next slide about actually introducing them truly to your team the reason that team exists why do you have a response capability there's probably a very specific reason by the way setup is

going to take multiple days right setting up the machine setting up your VM setting up your accounts organizing all your passwords all out of this stuff that's those the first couple days those are really boring and then you end up the first week with your sim training so how many folks here use the sim or having it work just yell at the name of it Splunk I hear Splunk spottier Splunk okay we got Splunk alright so cue radar our site now whatever you focus were using you want to introduce that to them in thorough detail the team origin was your team developed out of a reaction did you have a little bit woopsie in your network that you're responding to

right I know some folks who have gone from zero active continuous network monitoring analysts to having over 40 on a team obviously they didn't just think like oh we should probably have a really big team for no reason right so what are the real threats your company faces and what caused the formation of that team why are they really there what really brought them into your environment and then for that matter how are the team formed we've had it let me pull back for one second I've heard of I've completely not at my place of business at all I've heard of people working in this type of capacity on a sock for example four years at a time and then asking the

question what does NSM mean like what we haven't innocent team and they're like okay what do so how do the teams operate what are the capabilities what are their tool sets and then how do they integrate and that's all really important for them that can go back to the whole whiteboarding you know with this type of training you do a whiteboarding and you say here's our tiered infrastructure here's why it was set up this way and then you erase it and the next day you point to it I don't know whoever you say hey you go over there and draw it in the board and they go okay and then they try it and then they mess it all up and

they're like I suck do it again right so you just kind of keep fostering that idea oh the very bottom here it's very important folks know your capabilities what works well on your team and although this is controversial I stand by it 155 thousand percent the things that don't work well on your team the issues that you have on your team if you let people know about them upfront and of course a positive way then they can help focus on those things they're being integrated into the team to try to ease those things otherwise if they're not aware of it it could be the exact opposite so your sim what logs you have available why do you have those

logs what's the duration of those logs for you Splunk folks how long do they stay in cold storage before they go to frozen storage like what can they what do they have access to and on top of that where they not have access to that's critical say your company has packet captures and you say hey we can get pcaps and they're like oh cool well are you SSL offloading and for that matter how long do your pcaps stay in play like we can pull peak apps for up to 5 minutes ok awesome that's super useful right whatever so it's very important to let them know what they have access to and what they don't and then your tickets I

don't know how many times you know there's a lot I'm from the Phoenix Valley so what tech time for the Bay Area but I live in Phoenix now we have a lot of security teams but they're all very close-knit so we kind of know each other's tools and things that they use on a daily basis really and there's so many times people come in and they're not even just taught the basics of like hey we would prefer that you document your tickets like this like use these types of code blocks or like put you know we want this type of information or something like don't put ephemeral links because that's not gonna help when we're

doing ticket review a month later or something like that so that's just critical to kind of get up front and then again using example very common tickets as examples is super helpful okay networking in forensics anyone here want to come up on stage until I've learnt exactly how DNS works exactly how DNS works anyone I don't know how many was their hand over here maybe in this I don't know how many times someone's like I have a CCNA I've got a freaking degree I got a grad degree in networking you're like okay how does DNS work well you put the name in and then it gives you an IP okay well how's that how's that worked

what gives you an IP stupid like oh no no all right so actually understanding these things the core fundamentals those protocols is critical for the understanding especially when it comes to response the more data you provide them like a peek at the general pcap of a freaking DNS query couldn't like to spin people's heads so how do those work those pro calls your networking logs do you have a Zeke environment I hate that name bro I'm gonna just call bro right do using bro what are you using for your log parsing right how are those being bucket eyes and dumped into your system where they look like what format you can follow are they tab separated or the

comma separator the key value separated like well they look like alright and where can you go to get information on those damn logs email and again so many people are like god I've been tree as you can email for years now and you're like cool what does it mean when the field starts with the X - and they're like it's it's an X field ok what does that mean like would some stuff has X stuff like okay it's just an optional field right but why are those there and what uses those you know what's the user agent what's where they call X mail or excuse me an X - mailer why are those important how can use them for pivoting taking an

example email like a hundred lines and the headers and just going bottom the top can be like tell me everything you see you'll notice a lot of newer hires even with like a decent amount of sock experience we'll know like maybe a third of them if and that's being generous by the way and then we got Wireshark and pcap which is Network forensics coming up over here DNS is my favorite if you take away anything in this talk right now except for the fact that the power glove is bad and I really like these aside from that you gotta take away the following there's a video out there on YouTube it's called a cat explains DNS

it's made by a fella yeah fella in a nil Kitty phenomenal the guy really knows his stuff and so you a me he's eccentric as you can tell right so you're looking at him you're like it's really knowledgeable so I show it in every classic 15 minutes it's phenomenal but this goes to the example of asking your class you know like how does DNS work right and I'm gonna be quick here because of time I'm just gonna give an example of some things that we cover right you pop in an email or I'm sorry domain name up in the address bar what does your computer do it looks at the local cache and it goes the host file it

doesn't do that it reaches out your DNS server DNS servers most likely a local resolver the local resolver doesn't have the answer is gonna go where first to the teal these to the top low payment the root domains 13 of them made through em right if they don't know the answer they're not going to it's gonna send them over to the TLD domain so if you're looking for like farts comm it's gonna go I don't know where that go talk to the calm person right when it goes to the dot-com TLD they're gonna go like oh that's the authoritative server over there go get the answer when the answer comes back they're gonna get the

a record it's gonna have a TTL time to live if those are very very low and they repeat or change often I mean that could be something called fast flux so where are these things important like all these lower level protocol things are extremely important to Train there's a video of me writing on a board I like white boards too I can modify this you'd be white board something oh that's my project Wireshark a lot of people don't really understand how Wireshark even works right you have a peek at file what is it there's a bunch of zeros once that's all it really is right so how does Wireshark know to represent that data visually to

you well if you look at the top right hand side here we have an example ipv4 header remember headers Noor when you read about it in a book and then never carried on them again I do I don't care about them until I put them in a presentation like oh they're really important so you have the header over here on the left hand side there's an arrow pointing up where a shark is pointing to what I believe is the first blue line there I'm color blind blue or purple or something and it's pointing to the IRS's version for the actual value of which way over on the left is zero zero one zero zero the binary value

right you look down at the bottom left hand side or the other arrow is pointing down and you see hexadecimal value of 45 you're like well 45 is not 4 well as taking the first 4 bits what are 4 bits it's a nibble what's a nibble how many nibbles are in a byte it was 2 depending on the architecture blah blah blah this type of training is critical for people to understand how something as simple as Wireshark is actually show like visualizing what traversed over that wire ok so starting with the peak app and not just saying like this means a packet was sent back and forth but actually talking about the packet bytes here's the actual bytes here's how they

mapped to the header and that's how we're shark is then displaying the data to you starting at that lower level can be extremely helpful okay host base forensics again I typically recommend what I call week 3 and then week 4 host based forensics and reverse engineering if you have a fledgling team or folks with not a lot of seat experience I would take this out and kind of push them off maybe couple weeks or a month or two or something like that cool so in this case we want to talk about artifacts from your local types of operating systems if you're a window shop that's gonna be Windows artifacts starting with the file system stuff like

NTFS and the MFT parsing the MFT what's in the MFT what the user journal file is all that kind of stuff is very important registry analysis its windows so you know registry for you Mac users it'll be all your pee lists and their locations and things corporate forensics tools alright do you have a forensic capability like in case I'm sure you do because everyone does right so you go over any case and talk about how you can remotely connect to the disk and why that's important Erik Zimmerman's tools just google them you'll see why they're phenomenal the artifacts that you need how to obtain them parsing them and what that actually tells you just a loan you

could spend an entire day on like the ten tools on his page and then we've got deep dive into memory analysis some examples of artifacts in our classes we spent an entire day on this poster it's a sans poster it's hard to get away from sans content because they provide a lot of it and a lot of it's really good so this particular poster you just google the damn thing it's called Windows forensic analysis poster you're not meant to read that by the way if you're like that text sucks duh but if you just look at when you download it right you'll see all these different artifacts and it's like oh the artifact lives here

and when you parse it it tells you this and then going through each and every one of those is what we do if it takes a day okay if it takes a day and a half or two days ok whatever memory analysis take a look knead the water take a memory image of something like Stuxnet or black energy mid-sentence water there we go or something that's like driver base you know lower level Kitty kind of malware you stuff and just run every damn command you can find in the Volatility command reference do you never use volatility literally Google volatility command reference and then Google volatility example memory images and then with one of those names slides

available online by the way the example names I gave up there and just run every freaking command run every single one of them and every column why does it exist why is there a pig there why is there a handle there what's a mutex has a mutex different from a pipe all these different little things that like core OS concepts and fundamentals a lot of responders don't actually they're not familiar with that if they haven't worked or had that low-level friends experience good example of that in volatility there's a command called PS list stands for processed lists there's also one called PS scan PS lists will what's called wok the doubly linked list that's referred to in memory as the

eat proc structure and it just goes like from one to the next from one of the next and it enumerates all the processes that are shown in memory and it goes these are all the processes that are running right now there's another one called PS skin and that's because what malar can do is it can take that eat proc structure and simply unlink some or one or whatever of the processes from that PS list just walks that list so it just misses that process of memory PS scan is called pious scan because it scans through the entire memory image it looks at all different pool tags and it looks for anything as tagged as an e

proc and it goes like oh hell that's a process and it's in memory maybe that's running in memory and it will list them if you see an output difference between PS list and PS scan there's a good damn reason for that and so having your folks be aware of that is you know critical that type of analysis that's the type of stuff you want to show them like the differences between the individual modules and plugins why do they exist i hadn´t know how many people be like hey so they show the same thing it's just different names like ok cool story I'm sure reverse engineering the primary thing I want to focus on here carrier

files if you have a PDF and that PDF has JavaScript in it or if you have what we're looking at right here which is I believe this I think it was an imitate dropper I may have been wrong on that but our downloader excuse me whatever you have the visual basic that's in here if you've become fully reliant on sandboxes right say you use the Falcon sandbox or just virustotal they've got like 4 or 5 in there I think Tennyson does one of the ones is default in there or the names hybrid analysis is Falcon online you drop it in there's something like imitate that actually uses typically a loop where there's five different URLs and if you throw it

through an analysis sandbox as soon as it hits the first one it goes there's your call out and it provides it to the analyst meanwhile the other four because you didn't manually decode all the code that's actually in there you might not even see it so that's really if you look doing a meta-analysis by the way by just using sandboxes and if you're getting five or six okay you're probably cool but if you're getting one you're probably missing indicators Sam bone is a teacher here well not here here but San Francisco City College phenomenal fellow if anyone knows him here you know how great he is Sam's class dot info okay if you go to his website he's got tons of

curated content like odd ton whenever someone asks me where's one of the best places I can go for free training Sam's class taught info it's phenomenal all right so he's got am our course on here and we spend at least half a day just going through it so if you're like oh I can't get the URL the slides are online or just tweet me or whatever it's available finally the last week is working tickets thank you and working with intelligence so intelligence we start right off with iOS ease indicators of compromise what is a computer to indicator what is a an atomic indicator how are they similar how are they not similar why are they important so on and

so forth so a lot of folks are not familiar with iOS indicators of attack so we start to start talking about those and why those are important just seeing that PowerShell launched on a Windows machine it means nothing it launches on Windows all the time right however if you see PowerShell launch and then you see these four specific processes launch in a row and you see that happening on multiple hosts in your environment and that never has that could potentially be an indicator of attack so we talked about all those kinds of things and then we've got threat hunting wait for the banana I know the the dirt the d'azur thing right now everyone's like oh I

throw it hunting me yeah it's extremely important I was gonna ask folks here you know just shout some names out of what models you're following but everyone knows you're following the same damn model everyone's following miters attack framework anyone not following miners attack framework using something can you say what it is it's like okay all right well okay so any threat hunting is critical so we tried teaching that right up front using the Intel that you get from your own tickets your own alerts and then spinning that and maybe pivoting off that to find other related attacks or threats something as similar as an are as simple as an email coming in with a given subject and you search for that

subject over the past 30 days and you go like look our email plans didn't catch that campaign right and there could be 30 300 whatever emails and that one so then you go review that reinforcements final concept here I don't on the right it talks about quizzing I don't like quizzes in terms of you need to get a 70% or above I don't even want to score when it comes to quizzes but if you look into some of the learning management systems or the LMS is there's one that's like the king of all of them open source called Moodle I'm sure folks have heard of it here so if you grab Moodle and you make some quizzes I recommend making it

such that if you get it wrong it just goes like a dragon and I go okay and you look it up it's like that try again like they just just try to get them to just find the damn answer right and just put it in it's just it's almost kind of part of that whole rote memorization thing that we don't want to focus too much on but it's still good to have also if you folks have ever heard of Mnemosyne nemesis ii i believe i'm pronouncing that correctly I'm probably butchering it cuz it was a Greek god of memory or something like that there's an open-source tool called nimah seen in scene and if you look that up it'll it's

really really good for memorizing facts over time and it's really good for these types of concepts oh and then of course your tabletop you know you show them a file you say hey this came in and you downloaded off virustotal or whatever but you tell them I get this came in how exactly would you respond to this and then you kind of just go over it and and hopefully they didn't suck okay and that's it I heard we wouldn't have a whole lot of time for questions because I get yep yep yep I love to talk which you can already tell I'm well aware so we are using the SLI dot do with the code or access code besides SF 2019 I

don't know if I can personally actually answer those and a few flows from where that but we'll try to get those answered my twitter by the way is RJ underscore chap if you want to hit me up on there I love communicating and networking all kinds of crap so if you have any questions about the kind of training concepts or maybe a little more detail on some individual things just for sure let me know and yeah I also have two classes on Pluralsight comm there by the way to breakfast pong right great right there so if you want to trick those classes out we have some cards for free access so you can just check them out with that painting a

little self plug there but yeah that's it I hope you enjoyed the talk and thanks very much appreciate [Applause]