Red Team Engagements: How to Train Your Blue Team to Hunt Adversaries

cool all right uh [Music] so we're going to be sharing a lot of practical things that we've learned in our experience uh these things we think will be a help to any team regardless of where you are in your security program with your maturity uh we think it will add robustness uh add maturity for both your red and blue teams and it's going to equip your blue team uh with the processes that we'll share equip your blue team to be able to hunt whether it be the red team or an actual adversary um your experience may be different uh but we think it's really important to share uh things that we've learned uh the good

and bad along the way and uh pass those things along so first things first i guess just a little disclaimer uh you know the opinions beliefs and viewpoints expressed by the authors that being us are not necessarily the things that um maybe our employer believes or would necessarily agree with but with that out of the way just some introduction so again i'm brad richardson uh i'm currently an offset engineer on the left you see i tweeted richard jb and i write the occasional blog post at medium i like to include things i think that will be down to earth helpful wherever you are in your walk in security that you can use on a daily basis

and uh i think it's fun when i have time to write the occasional usually offset related kind of tool my most recent tool which i'm going to be publishing some updates for is called slackhound it's a reconnaissance tool for red team but blue team could use it also uh you think of a reconnaissance tool for slack workspaces uh you know it's common to to a few years ago red teams would target active directory uh learn just about everything from the organization from dumping active directory domains but today you can find a lot of that information titles people phone numbers uh and much more in swac workspaces so check it out and with that i'm going to let one of my best buds in

collee introduce himself and walk you through the next few slides

thanks brock good morning everyone uh i'm motherboard i work as an offensive security engineer karma on the right side what you see is my twitter handle i sometimes write blog on the medium post sometimes develop new tools you can find them on my github page or on my docker app repo uh the most recent tool i wrote with brad is gcp hard it's sort of an offensive toolkit for google cloud platform and i'm a huge fan of atomic write team i've been using it for a couple of years and i have the pleasure to contribute some of my unit tests to atomic rectum get up repo so if you're not familiar with the project i highly recommend you check it

out all right so let's start this conversation with a fun brain teaser dexter morgan is he an advanced persistent threat or is he an incident responder so show of hands for those of you who think he's an apt all right show off hands for those of you thinks he's an instant responder all right so i'll tell you a little bit of what i think you know i had the pleasure of watching this show twice thanks to the pandemic second time after i joined the infosec so dexter loves to do red teaming in his free time but he gets his w2 for working in ir however since he has that attacker mindset it makes him very good at what he does

uh you know when it comes to investigating crime scenes see so normally if you are familiar with the show he walks up to a crime scene he looks at the blood spider pattern uh figures out how the crime might have happened looks at the crews clues around the crime scene to ultimately track down and hunt down the attacker so dexter is an apd works in ir and that is what we want from our blue team obviously we don't want to hire apds to work in ir that's not what i'm saying uh but what i'm saying is we want to you know arm our blue team with the capability that they get us alert then they investigate the alert to

identify a chain of events as they would have happened to ultimately track and flush the attacker out of the network so for example say there is a high fidelity pass the hash detection that just triggered so we wanted our blue team to be able to do the following one identify whether the host that attempted past the hash is initial point of compromise if it is not patient zero how did attacker laterally move to that host and find that initial point of compromise all the subsequent compromises to ultimately flush the attacker out of the network that is what we want from our blue team obviously it's easier said than done but it's difficult not impossible so as brad mentioned you know we'll be

talking about a maturity model on you know how you can continuously and practically train your blue team you know to be good at detection and response we broke down that maturity model in four phases and i'll be using a little soccer slash football energy here so you want to play 90 minutes of soccer game right the basic requirements are you have to be match for it so you can run around for those 90 minutes and you have to be you know good at you have to be continuously practicing those basic skills like passing shooting tackling and dribbling so the first three plays that you see detection chart purple team exercise adversary detection pipeline those are

the phases that it trains your blue team for those basic skills and become make them match fit and edward serial detection sorry adversarial services life cycle which brad will talk about is that friendly match between red and blue that prepares your blue team for actual match against actual adversaries so without further ado let's jump into phase one detection chart if you look at the detection chart uh it feels like four different paint buckets exploded on miter attack however there is a method to madness so let's understand what is detection chart detection chart is a visual representation of how good or bad you are at detecting attacker techniques on the scale of zero to three so for

example say you wanna say you take an example of pit's admin which is t1197 in minder attack if you don't detect that gtp in your environment at all that's a score of zero color red if you do detect that dtp but not its variation so certain util is one of the variations bits admin like if you put carrots b i d s your av or edr tool can mess it that's one of the variations if you don't do the variation that's a score of one color orange if you do detect bit segment and its variation that's a score of two color green and maybe bits admin is not applicable to your environment because you're mac only sharp you don't use

windows machines at all you're one of those unicorns then it's a score of three and color gray so you know when you look at the detection chart that's what those color represent and it's you know sort of nice way of you know nice visual visual representation of where you need to focus on now normally at this point i tell you that you know you want you won't be able to do this for every single ttp in mind attack because there are so many ttps so you have to know your environment and combine it as your experience as a red teamer to prioritize the list of ttps that you're going to test this for however there is a tool out there a gamify

attack by adam machinkey it's on red canary get up repo that tube is basically a crowdsourcing way of prioritizing ttps in your environment you can send that survey to your team and you know you can collect the survey and have you know everybody chime in on which gtps you should prioritize to test for so feel free to check it out now you understood what is detection chart let's figure out how to build one and uh it's it's pretty straightforward you will need gold images of the you know different operating system you use in your environment so vms for windows and linux and maybe a macbook for macbook gold image for mac os once you have those gold images

you just have a prioritized list of ttps you pick a ttp from there you go ahead and collect the unit test this is where atomic right team can be your friend it can come in handy but obviously you want to add your custom test as well to have more coverage one thing i want to note is automation will be your friend uh the more you automate the more streamlined it will be going forward then you run those tests on those gold images and then you validate if the alert is triggered or not now when we say a detection for the purpose of this presentation what we really mean is an alert so if you do find the ioc is in

the logs but it doesn't really trigger an alert well we don't really consider interaction for the purpose of this presentation so just think of it as detection alert alert detection and when based on whether the alert is triggered or not you score it a zero to two on the previously on the criteria i mentioned in the previous slide and you once you do this for all ttps that sort of pills uh that detection chart for you which will be basis for the phases coming forward and the next phase which is the purple exercise now before we move into next phase uh i'll talk about a dual line route called detection navigator i used to do this in

a spreadsheet however it is very cumbersome micro comes up with a new ttp a whole new technique or tech tech it's hard to keep track of so i wrote a django-based web server that is supposed to make it seamless and i've also included atomic test automation scripts that you can run on your mac or linux machine to test your coverage but feel free to add your own tests on those scripts i want to talk a whole lot about it here in the interest of time but that's the get up page for the tool and there's a blog link in the readme file that you can refer to for more information so at the end of this phase you would

achieve following you would have a prioritized list of ttps prior translation to your leadership on how good or bad you are at detecting those attacker techniques and you know at the end you will also have performed a gap analysis that you know sort of tells you which direction you want to focus your resources on as a bonus you can use this method to value your ddr tools so mighty does this you know vendors submit their tools to mitre miter picks the thread actor group and against those thread active group miter you know scores the cdr tools however if you're doing a poc on adr tool that miter hasn't evaluated or maybe might reuse the thread actor group that is not

relevant to your environment might evaluate a threaded group that is windows focused but your environment is mac heavy well you might want to go with a different thread actor this method you can use during the poc phase for you edr evaluation phase two purple team exercise for uh in this presentation uh you know we'll not do a whole deep dive on how to do purple theme uh right but we would just rather walk you through what it would look like to do a purple theme exercise if you are interested in a deep dive talks check out this talk from cetera cohen's uh he talks about how to do purple leave exercise in a non-active directory environment but you can easily

translate it to active directory environment as well so let's see what the workflow looks like now phase one you build a detection chart you know which gtps you want to build detections for you pick a ttp from there you already run unity unit test in the last phase so you would have an idea what kind of indicators are generated based on what kind of iocs are generated you either work with network engineer system administrator or an application owner to get those ios logged into your sim then you'll work with the sim expert to build the sim query that will trigger an alert on those iocs then you'll test and review that sim query we'll talk in detail about how to test

and review the sim query in the next phases but if it meets your standard for testing and reviewing then you'll deploy the alert to the production so let's make this all all more digestible using an example and what better example to use than dc sync [Music] absolute favorite attack for any red teamer out there and absolute worst nightmare for any company because you have to rebuild the domain once this happens by an if it happens by an actual attacker so for those of you who would have run dc sync for on-prem active directory controller you would know that it generates two iocs at a network level and at host level so if we look at this

workflow we picked a dtp that is credentialed dc sync and we identify the ioc generated are at network level and at a host level so we'll now we'll work with the network engineer and domain admin to have those ios iocs logged to sim then we'll work with the sim expert to build the sim query that will trigger an alert for those iocs but exclude the domain controllers from it test and review that same query and if it meets our standard deploy it to production obviously i'm making it sound this very easy it is not so i'll talk about you know one hypothetical challenge each but depending on your environment you might run into different challenges

so at a network level detection your network has to be architected in a way that all your traffic to domain controller goes to you know that ids that is sending the iocs if it is not you get potential blind spots and at a host level detection whenever your domain admin team adds a new domain controller you want to have a process in place so that you know your blue team or detection engineers get notified and they can exclude the new domain controller from that alert otherwise it is going to you know flirt your bluetooth with the alerts so these are just the hypothetical challenges you might run into but depending on your environment they might

be different challenges depending on which gtp you pick they might be different challenges and you know that is why detection engineering is difficult sometimes you don't have the loss sometimes you don't have the visibility sometimes gpo is overriding what you're setting there are just so many issues that you will come across during this phase but you just have to power through them if you want good detections so at the end of this phase you would achieve following you would have improved detection capabilities this is a recurring process to validate detections all the phases we'll talk about here are recurring phases they never really end it's a sort of that infinite mindset jason was talking about

you'll just be visiting at different point in time depending on where you're at and this forces your red team to improve they will not be able to use commodity ttps meaning they'll have to come up with their own which will eventually make your blue team better and as a bonus it provides insight into the effectiveness and value of the security tools you are using right we all know the sales pitch for different security tools uh and we all know sometimes how they turn out when you actually start using them well say hypothetically your ids is not capable of seeing those dc sync logs well it's time to change the ids and for the next ideas that while you are

evaluating you want to put the put that question in the questionnaire that you sent to vendor so that you get the product you need to be able to build the detections you want [Music] phase three adversary detection pipeline this phase can go parallel to purple team exercise the main difference is purple team exercise the red team involvement is very heavy uh not so much in adversary detection pipeline maybe just in review phase this phase is designed so you know you can make your blue team self-sufficient especially when your red team resources are limited and it is also designed so that you have a documented trail of the detections you're building and we'll talk about why

and you know in couple of slides but let's look at the workflow first uh based on the detection chart that you have the detections you for the ttps you need they go in the ticket backlogs of whatever enterprise ticketing software you're using servicenow jira then your detection engineer either gets assigned a ticket or they pick a ttp they want to work on either way then there to look at the ttp the ioc is 800 make sure that those ios get logged in sim then come up with a logic to detect those iocs and then build the same query that represent that logic that's your in progress space once they feel confident that their sim query they build catches the uh you

know malicious activity they'll go to the review board and this is the review process that we were talking about in that phase two slide from test and review in the next slide we'll show what the questions look like from the review committee but if review committee finds the answers to those questions satisfactory the content gets accepted gets deployed to the stem if review committee finds something is lacking well the detection engineer goes back to working on the feedback and comes back again when they have made the improvements so again let's make this more digestible using an ex using an example and this time we'll use a different example uh detecting ssh password spray so based on that workflow that is password

prevented ticket backlog now say you got sherlock holmes in here blue team right who doesn't want sherlock holmes and their blue team the best investigator however i would definitely want him on my blue team now sherlock holmes in a blue team gets assigned this ticket and sherlock looks at the ttp it's like ah this easy you know generates a network level i see oslo livesc and comes up with a logic to detect that attacker technique that you see on the right top side at network level sherlock is like all right multiple ssh connection different destination hosts same source ip and at host level multiple failed sage logins different username same source ip then sherlock builds the same query to

represent that logic and goes to the review committee before sherlock goes to the review committee he should have answers to following questions ready one is there a run book for this alert so that the junior analyst can follow the steps if the alert is triggered two what is the proposed maturity of the detection is experimental or stable you can have more category if you want but if this detection is experimental sherlock has to work every single alert until it becomes stable the other two questions are designed so that we don't you know create any alert fatigue or waste anybody's time what is the how often will it fire and what is the false positive rate we don't

want to flood our bleeding with alert or waste our time and the last two questions are where the red teams can chime in which is is there an alternate scenario that can trigger this alert so if you look at the network level logic a simple nmf scan or 22 will trigger that alert and when it comes to response context is crucial a response to a nmap scan is very different than you know a response to ssh password spread you have to start looking at successful ssh logins if you are dealing with ssh password spray so context is very crucial and that's why it is good to be aware of alternate scenarios and the last one is is there a

way this logic can be bypassed you'll not always find a silver bullet for every single attacker technique you're trying to build detection for if you do that is great but if you don't you want to be aware of you know the loophole so you can put some mitigating controls say the review committee accepts the answers to these questions content content gets handed over to the sim expert who will deploy it uh for alerting and the sprint will be complete so at the end of this phase you would achieve following uh you would have agile content creation process i know i'm just trying to throw out big words like i'm some software engineer where it is indeed agile

uh create self-sufficiency for a blue side especially when you have limited editing resources as a red team you don't want to be bottleneck for detection engineering for your blue side and documented trail in whatever ticket enterprise ticketing software you're using servicenow jira especially important if somebody who built the detection left the company and you want to go back to either audit them modify then tune them whatever reason is super important so now that we have become match fit and you know practice those basic skills are dribbling tackling passing and shooting i'll hand it over to brad for that friendly match between red and blue [Music] so what you see here is what we refer to

as the adversary's adversarial services life cycle if you look to the left you see where red team comes in with our very critical planning phase uh is going to ensure a rig a really good red team exercise the red team starts with the planning of the exercise uh when you move into the execution that'll be that friendly match between red and blue uh after that you're gonna see uh aars up there um we're gonna talk a little bit about each of these steps but for now just if you're not familiar familiar with this concept aars is really think of it like an improvement plan it comes from the government world and military when they conduct exercises

and then out of that we'll have more of our reporting in our remediation this is really where red team can add a lot of value and steer the remediation efforts to drive that continuous improvement in your security posture we'll talk a little bit about what you see on the left uh but keep in mind now for now um we finish uh as red team we're gonna come back at the end of our uh different steps in our exercise and we're going to do a blind test to make sure that all those good detections that we built or improved work the way we want them to work triggered when we want them to trigger and we're going to validate that we have

a good robust response if we just had the detection piece and didn't have a solid response then we wouldn't be complete [Music] so jumping into each of these phases we'll talk about the prereqs in our planning again this is really critical from our viewpoint so what you see on this slide on the left are the things that we think are key and on the right we have some basic examples of how you would implement that so what you see here is we're going to start in our planning with getting feedback from our leadership our executives wherever we're targeting we want to make sure that we take those concerns into consideration design those into our goals into our

objectives we're going to use that as part of the basis of our rules of engagement this document is going to be a lot of guidance around what we'll be doing during the exercise we're going to take that once we have it baked the way we want to our legal team this is going to add transparency to people outside of security as to what we're doing it's also going to help keep the red team out of legal trouble we're going to also establish a timeline for emulating the different phases that we're doing as a red team and we have an example here what that looks like but the main thing to keep in mind is this really helps us make sure

that whatever we're planning to cover in our exercise whatever the findings are that we want to get out of our red team exercise um whatever ttps we're planning on using uh this is it really ensures that we don't get stuck in a phase sometimes you'll hear the term uh time boxing uh this just keeps the exercise moving forward so uh if you think well we meant to walk through four different phases of miter attack but we got stuck in initial access we couldn't get a phishing payload through what do you do about that so that the exercise keeps moving forward and one way of doing that is laying out your your timeline so uh also

if you're familiar with the term white cell this is really a mediator if you have in your security team the resources to establish a white cell a person or a couple people on the team that can be sort of that neutral mediator between red and blue this can be really helpful it takes load off the red team when red team gets questions um especially during deconfliction deconfliction is really the process of just uh was this red team or was this some other uh anomaly that was triggered an alert and uh also uh y-cell can be really important when it comes to reporting maybe white cell remembers things completely different than red and blue did so you have that

neutral outside observer that can help keep things straight as well if you have a green cell and can establish this they manage the red team infrastructure and if you're like using a cyber range it could be any other special hardware software that you use during your exercise it's been our experience that white selling green cell are luxuries a lot of times there's nobody that's really available to do that in the way that it needs to be done if you if you have somebody on the security team or inside your company uh they want to be white cell but maybe they only have a couple hours to devote a week it's been our experience that's probably

not going to be enough it turns out to be more of a full-time job than just uh attending a daily outbrief or a weekly outbrief it's actually pretty intensive to do it right one other thing that we think is important in the planning and prereq phase is deploying infrastructure especially like c2 infrastructure anything that your red team is going to need um the key takeaway here is uh you don't want to get into the execution phase and decide that you need to set up you know c2 infrastructure in aws or or somewhere else or anything um that could have been deployed uh in the planning phase things are always going to pop up there's always going to be things that

change your attack path and maybe you need to set up some kind of redirector whatever it might be but as much as you can plan this out in the planning phase it's going to save you time when you get into that execution and the engagement or assessment starts the clock starts um so keep that in mind as well and then just looking quickly at some examples of what those uh things are we get our objectives uh that might be like we want you to start from outside the network and work your way in see if you can ex feel sensitive data whatever it might be uh it could be like uh you're going to do spear phishing but

you're not going to do spear phishing for certain people in the company might be considered sensitive uh put that in your roe this is what we're going to do this is not what this is what we are not going to do your target network range and then another key thing is we think it's important to lay out if you needed to stop your red team exercise how would you go about doing that let's say an active attacker hopefully this never happens but you find one on the network red team's always getting into hosts they're doing reconnaissance what if your reconnaissance shows that you find mimi cats and it's not yours what do you do to stop the engagement

and alert everybody escalate and get blue team looking for the actual attacker so maybe bake that in as well into your roe and then again about the timeline you just want to lay out methodically the steps that you're planning to walk through during your engagement so it might be initial foothold and how you're going to do it um it might be 8th the 14th you'll do that 15th through the 21st you know you're gonna be walking through the other phases but the idea is to always make sure you have your plan laid out that keeps you walking through the exercise so the execution of the red team engagement again you see on the left things that we think are really key

um strictly follow the rules of your engagement um whatever you say you're gonna do try to do it and whatever you say you won't do you know if it's off limits stick to that as well um another thing that we think is important and would call out very practically is uh red team shouldn't do anything that would obstruct blue team's ability to investigate you know it it could seem cool to turn off edr tools or take down splunk um but if the blue team can't investigate um it kind of defeats the purpose of you know what you're trying to get out of the overall uh exercise i know in later maturity sometimes you might do something like

cyber deception those are to me very advanced uh things uh usually require a lot of maturity that probably is difficult for a lot of teams to achieve so just generally we don't want to do anything that would hinder blue team it's a friendly match and if red team's not getting caught easily see what you can do to increase the level of noise put some things out there that could be created as ioc's that the blue team can hunt you by catch up keep that match going well for both teams and then i talked about the after action report and again it's kind of like an improvement plan so it doesn't have to be you know really complicated but the

idea is to bake in this documentation layout while it's fresh you know what went well what didn't go well what can be improved and how do you want to plan for future success so you know just a couple of examples there if for the red team they deployed some redirectors that was located uh geographically and from an ip level view that looked like it was close to the company maybe that helped obfuscate some of the traffic maybe blue team ignored some of that traffic because it you know looked like sign-ins from um you know close to home if that worked well call that out blue team whatever your wins were there you know it might be correctly identifying

an smb password spray call that out if it led to a compromised laptop which helped you track red and it also showed that you know the network segment that you built the way it was designed putting that particular team or organization there was effective call it out you know as well as what didn't go well what can be improved uh all these things will provide that documentation you'll remember more about what happened in your exercise and you can carry that forward to the next exercise so we'll talk a little bit about remediation uh remediation to me at least is uh not really much fun uh it's usually taking unplanned work to other teams uh would you love that

and it's also a way that we as security and especially the red team can put our signature on the things that we think really improve the security posture uh security orgs are all working in a hundred different areas but this is where we can say you know we demonstrated that this is an actual risk and we know that it's an actual risk because here it is here here are screenshots of it so we can tailor a lot of the remediation work that's being done red team probably knows better than most other security orgs what causes attacker pain because we're playing the attacker so what causes us time and stealth everybody's equal when it comes to time

and this is where we can say if we did these things it would really slow down or hurt an attacker's ability to hurt the company so we break those down into prevention and also detection and response we're focused on all three in our exercises for us we categorize prevention as anything it denies degrades and disrupts an attacker's tactics of course again costing them time and stealth so you can think about this in terms of like your normal preventative security controls like mfa great security control network segmentation could be even like stronger passwords or stronger password policy uh all great security controls uh detection response you know this really uh involves the collection monitoring for bypassing the preventative controls as

well as you want to think about things that um maybe can only be detected right but we can have a solid response if we can detect it so motto talked about dc sync not a lot of great ways out there to prevent a dc sync but we can detect it and if we detect it then we can take action on it and again mfa great security control great preventative control but we all know that there are bypasses out there for it and then of course living off the land type things so abusing legitimate accounts in in cases of admin accounts these are things that we can detect even though it looks very normal and then we can go after it

so again kind of back to that adversarial services life cycle in these recurring phases we shift a little bit from what the red team is doing uh red team is still involved we talked about the reporting and remediation coming out of that the ttps that the red team used in the exercise during the assessment those are going to be a lot of the basis of how they uh form that color on the detection chart that you saw earlier and one thing to note here is when we talk about the ttps now maybe they map really easily to miter attack or maybe they're custom maybe you're really mature team and you're doing a lot of custom tactics uh writing your own

custom tools in out of the box signatures things like that just aren't working well it doesn't really matter for our workflow whether it comes from either attack or whether it's completely custom it's going to feed in it's going to be the basis of our unit test for the detections that we're going to build to get them down the way we want triggering with a low false positive rate and of course either way they'll feed into our detection pipeline and when we do those unit tests and we sit down collaboratively as a team after our exercise uh red and blue together making purple uh we will get those uh and ensure that maybe it was just we couldn't detect it

at all maybe we could detect it a little bit but not the variations once we get those detections down the way we want them then we'll also ensure that we have a strong response for when they're triggered and then later red team's going to come back in and just do that due diligence with a blind test to make sure the detections really work the way we thought they would and the response was really the way we wanted it and just a thing to call out here this is a really good area of opportunity for you to take and measure improvement so you think about when you do the red team exercise maybe you didn't detect

anything the first time maybe you detected a little bit maybe detected really well but response was weaker than you expected you know you can measure in time again you can measure your improvement by uh how well you did after you built your detection and build your response i think this is really practical it's a way of reporting on just that incremental improvement and if it's not the way you want you go back and you do it again so at the end of this phase um it's really going to provide a lot of documentation reporting you can start to build metrics uh that you completed in these phases uh from one two and three um it's going to

help your blue team learn to really proactively engage now it might be the red team now but it can be any attacker adversary and along with that engagement they're going to learn to build iocs and become better hunters and then again in the remediation phase this is where we really get to ensure that all the effort that goes into remediation is what we think is going to improve the overall security posture and that comes from stronger prevention detection and response controls and that's the basis of what we're doing so in closing with that um you know we we really believe from our experience that blue team can use this uh as training and be better proactive hunters

for example if an exploit is released let's say it's for a zero day it doesn't really matter at this point uh blue team is used to looking at whatever the threat is whatever they're uh going to be hunting and targeting uh building iocs from wherever the content uh originates from and building those iocs building detections and building the response to it all the phases are methodically linked as we said recurring and they provide the basis of how you you can incrementally and continuously improve your security posture and then finally transparency to leadership so we talked about uh the remediation your leadership your management they're going to see with lots of documentation how your security controls are improving

uh they're going to see all the great work that red and blue are doing together they're going to see how the teams are functioning uh better and they're gonna see the overall or have a better window into the overall security value and bigger picture uh as well so we talked about the aars and we talked about the detection chart and that you can think of the detection chart before and after um they're also going to see the communication that's happening um in the overall reporting and with that you know we believe that they're just going to have a better idea of all the good work all the good assessment findings coming out and all of that improvement that's

happening and what they're getting for their security budgets and so with that we'd really like to thank you for coming out to our talk and a big thanks to besides rdu um and if we have any time we'll be happy to take questions and if not we'll be happy to meet afterwards [Applause] foreign