Ryan O'Horo - One Hundred Red Team Operations A Year

thank you I thank you all for coming and I appreciate besides Augusta for bringing me out here I started my journey in red team proximal to besides Augusta several years ago and I gave my first talk here on transitioning from ped testing to red team and it was also very grateful to give that talk here so this presentation presumes you have a red team already big or small and it doesn't explain how to start a red team I must say though that the only wrong way to do red team is to ignore the needs of your blue team now we are seven engineers and one manager we have grown and changed in size and shape over the years we've been

reorganized and shuffled and we all have various backgrounds and levels of experience we work alongside our team and we encourage both casual and formal collaborations we even eat lunch together together and every day we share cool stuff with cool people people who are engaged and challenge us and make us want to make each other better now we support and are supported by our incident responders our detection engineers our security engineers even our leadership and we're also supported by our internal business teams so we collaborate with product owners and business teams to understand how they get work done and how to improve their defenses against threat actors now we're distinct in function from our penetration testing and vulnerability

management teams combined they are both four times as large as we are and therefore Minh Spurs ents prevents us as a red team from having to spend a lot of time hunting for and reporting on vulnerabilities this also allows us to focus on detection response and threat actor emulation so our environment spans some 1,900 physical locations and this allows for ample opportunity for rift and variation and failure in the environment our security technologies and some thousand manually maintained rules to support those technologies are spread all across the enterprise their functions are maintained by a variety of teams including vendors and IT teams like desktop engineering one team may install a technology another may maintain it and yet another may make

sure that it's actually running so the equal distribution and effectiveness of all of these technologies has to be reasonably assured right our red team is a major driver of that goal sorry I'm rushing through this a little bit I'll give you some breaks but I want to make sure that we get in time here so in service of this our red team gathers information from inside the organization order to the best prioritize the playbooks that we're going to use for attacking the environment the business deploys new technology at an astonishing pace so we're constantly learning about those things we combine that with data collected by our extremely capable threat intelligence team and again we'll use that to pick and choose which

techniques and behaviors we want to exhibit on our red team operations Japanese macaque break like many other red team's we were doing a handful of long-term operations a year however we found that there's a finite amount of data that we can extract from individual operations told the affected hosts or the number of techniques that we use what was kind of surprising was that even the individual analysts assigned the response for those operations had slightly different approaches to their work we found that the greatest variability comes from the incident response process itself rather than from the environment you'd think that in a very large constantly changing Network those things would be the king of the unknown the environment itself it's a

Wild West but alas the response process actually provides us the most opportunity for improvement so we came to the conclusion that if we want to maximize the amount of novel information our red team can create will run shorter more frequent exercises we also acknowledge that short operations have a few disadvantages among them some threats may persist for months inside of the network before discovery adversary actions can become complex and have a large blast radius accurately simulating those threats is difficult with addict without adequate time so in order to achieve an ambitious goal like 100 red team operations a year we first organized two weekly operations a year these had similar structure such that every step of an attack was covered in

both cases but the key difference was that one operation always completed with a technique that matched a detection rule yes this means that operation would always intentionally fire an alert now I hear you screaming colours at me and that's just fine just sit back and relax we'll get through this what this allowed us to do is to scope certain operations almost entirely around the response process we can then scrutinize the incident response process rather than add compounding factors around detection and vulnerability sorry visibility in the environment the second weekly operation was more typical of other red team's done mostly within visibility or detection gaps in order to achieve specific goals we sought to answer questions like are all existing

rules generating the expected relates from endpoints at all Network locations in order to do this we had to add a third weekly test a hands-on QA test and that at three projects a week let's start and add up now and so this meant that we were on the hook for manually doing about 150 reportable exercises per year and that number even now is exhausting just to say so how do we go about this beginning each operation with an earnest attempt at initial access we will have expended a lot of time and energy on a small selection of tacker techniques consider this list of technique categories just from the miter frame attack framework the value of our

operations is primarily in techniques that occur after initial right this is where our most impactful improvements can occur given that understanding our most frequent operations are executed with the assistance of volunteers from inside our business those volunteers usually help us in executing a social engineering attack we on the red team will create a scenario like an email and that volunteer will be asked to do what people are so good at doing opening emails this work is also commonly done on check out hardware which is lent out to the volunteers specifically for testing purposes and since our red team operations are so frequently responded to the hosts and accounts scoped into those operations are sometimes disrupted

by the incident response process so we're having to be cautious about how frequently we interact with production assets and productive people which is also something that's just as important as production assets however for us only a modest percentage of the hundreds of operations we've done have had some business impact and those impacts are almost entirely in end-user productivity and this an example this would be like if we were resetting the password of on a user's Active Directory account so we've essentially traded some demonstrations of high risk techniques in sensitive production environments for far more frequent demonstrations of low and medium risk techniques in production like environments we get additional benefits in engaging more directly with

people within our business and doing so in a positive way where we invite them into the process bringing awareness of what red team does outside of the security organization snow fox play break so taking an ad hoc approach to development and infrastructure was really holding us back it was taking days to prepare the components necessary to run an operation which meant that a weekly test was running out well beyond the test week itself and with multiple concurrent operations a setup where you servers that live forever but get constantly changed and repurposed over time was preventing us from scaling so we changed our approach to infrastructure by doing a complete tear out we built from the

ground up a brand new cloud environment we're computing resources were automatically deployed as needed for specific operations and then destroyed after their completion and this excuse me this deployment process takes minutes for us this was achieved using a combination of infrastructure and configuration automation deployed from code repositories and configured with a very small text file well we have a single master branch for everything all of our team members can deploy their own branches or make changes before or after deployment in the environment now what we've managed to do with this is drive down the busy work streamline tool development and let our operators focus on executing techniques which is that what the core of our red team does in

operations so if you've picked up when I'm laying down we do DevOps on on red team and I would suggest that all red teams do DevOps in one way or another if you manage your code and tightly knit it with the underlying infrastructure you're pretty much doing DevOps so let's think about that methodology for a little bit we're also planning our development out as best we can we're tracking everything in issues using project management tools but the security of our environment is actually the very first priority for us and since security entirely builds around automated configuration we can worry less about errors made by operators now this all may sound like a lot of overhead but be sure that we're very

conscious of how much time we spend making our tools and now if I write a piece of code I have to make sure that someone who's never seen it before can run it safely during an operation and if they choose to make changes and commit those changes back to the repository this means everything has to be very well documented now with this process what managed to do is to make tool making sustainable for our red team now maintaining detection awareness has been pretty challenging for us there are dozens of distinct technologies in the environment that our red team has to keep track of where detection and Prevention reside and so as an organization we don't even own some of

these technologies it may be hard or impossible to query a rule setting it exists in a particular technology for that very reason but whatever can be made available to Red Team has been made available we operate with complete technology transparency and let me let me emphasize this saves us so much time it also gives us opportunities to spot problems that thousands of Red Team operations may never reveal our blue team is built out a tool which catalogs all of the rules are in our environment along with metadata about when they were created where they live and what intelligence they were based on now we use this in several other tools to pick our techniques while print and planning

operations our Ridge can reach into these tools to a search on the keyword and find out most of what we need about how we as an organization can detect a particular technique now our red team and our blue team also share the same physical space and that allows for casual collaboration for us we share a chat room specifically for red blue collaboration where we can talk about detection techniques since we do this so frequently we've actually we've had to keep our reporting requirements to the bare minimum and what we do is with each weekly operation we create a very consistent wiki entry with where the reports are completed will will package them up into an email just copy paste

and then we'll send that out to actually a fairly large list of security team so that they can all be aware of what red team is up to now for our long-term operations we they may get a longer report but for the purposes of the majority of our operations we may be a twenty minutes working on our reporting so this is the template that we use for weekly operations at the top is a short summary of the operation and its objectives and we identified the subjects of or subjects of testing and we also know what role are accessed within the organization that they have and that's in order to help us put the risk in perspective in the organization

so we'll document details about the delivery mechanism and put a time stamp on everything of course this includes when the compromised activity begins and when it ends if there's exfiltration performed we'll identify what that data was where it was staged and how much we'll document specific I OCS host names addresses email addresses hashes URLs that type of thing and to tie our operational work to actual real world threat actors will list out how each attributable technique is related to in the wild threat actor and this will include a link to either public or internal documentation on that threat actor or technique will enumerate any alerts that may have fired or should have fired I might say and then of

course we link that directly to the case or in our case management system this may seem light on the details but keep in mind that we also keep logs for all of our Red Team tooling right so if we need to go back and look at a command or a process we can dig back into the database and pull that information out if necessary during a debrief or if somebody has specific questions so with a high volume of operations the opportunity for unnecessary escalations increases pretty dramatically when an operation begins what we do is we tell a selection of partners within the blue team what the timeframe the operation is and who the intended targets are there's

an agreement to keep these data points secret from the people who actually respond to cases and what that allows us to do is let Red Team operations run their course but allow the right people to de-escalate if necessary burden snow Blake great okay so eventually the grind of running this many operations three per week is with only a handful of engineers was really getting the best of everyone not only the red team but also our detection engineers or incident responders everyone was overwhelmed basically underwater with the amount of information we were gathering so our term operations were also suffering from lack resources and we weren't getting good value out of that manual QA test so

we came up with our ambitious plan to conquer improve through brute force basically look like this though perhaps we might take what we've learned doing to labor intensive operations a week and a third QA test and incorporate them into a more balanced schedule to approach this we now separate operations by time span as well as define different objectives for each of them we take some of that work that we were doing is in through manual QA testing and we automated for us long-term operations are strategically focused looking for gaps in process policy or design medium-term operations are centered around detection engineering capability and also working with individual business teams we now run to these medium-term operations a month short

term operations primarily exercise our response capability and do some quality control on detection engineering we cut those down to once a week so now that we have wide coverage of different organizational goals we can maintain high quality and quantity of results with a schedule that looks a little bit more like this everyone can kind of take a breath now with automated QA testing we can monitor the effectiveness of detection and prevention technologies within the organization without the red team being the first to find out about it our red team partners with other security engineers to develop this capability and we're hopeful that we'll be able to run hundreds of tests daily that assure that there's no human

interaction required to make sure that things when things fail or things drift in the environment we'll be notified automatically about it now not only can we provide high quality intelligence back to Red Team operations but a red team is not burdened with continuously monitoring for drift or failure in the existing controls now it's we call continuous improvement is dedicated to security measurement and accountability this team tracks security performance across multiple groups and is the primary facilitator in the red-blue relationship after each operation the data that both red team and blue team generate are compiled in an after-action report and because this report is generated by a third party we have the benefit of a more objective

viewpoint then that report is reviewed in a meeting with members from all of our partner teams the debrief and there's usually 20 or so people in the room when this happens the entire timeline of the operation will be reviewed and reassembled and everyone in the room and those debriefs will look for opportunities to improve continuous improvement also sifts through and reports on large amounts of security data for example here's an analysis of four years worth of our case information this chart illustrates how the weekly volume of malicious cases correlates to the average time it took to contain those threats what this is suggesting here is that there's no direct correlation between those two things right where we see that if the case

volume increases our contained time remains fairly consistent so if the the dependent factors here are yet unknown and require further investigation continuous improvement also regularly meets with our management to discuss overall security performance and red team back data this this very data is a strong contributor to those discussions now we found only a few ways to quantify red team's performance and the most important to us seems to be the number of red team generated action items we estimate as long as red team is identifying areas for improvement and those findings are being addressed we are hitting our performance button benchmark one of the metrics continuous improvement tracks is where each action item lands in terms of how the action

might dressed now this is a breakdown over a four month period of all the action items our red team generated we can see that visibility is a tiny slice there some time ago that visibility slice might have represented a larger cross-section however as our red team has spent time operating in the visible area those gaps were greatly reduced so we're making sure that our red team is addressing the area's most in need of improvement we can also examine red team performance using a containment time metrics so this is data for an entire year and we can see that containing our red team often takes longer than the average melissa's case but we can also see that we're really too far ahead for

too long tracking this data helps red team tune its operations to match the blue team's capability we are maturing alongside the blue team not running far out ahead where the Dragons of diminishing returns live another way to measure red team is frequency frequency of operations meeting established goals there's rarely a shortfall for us here because our goals are fairly straightforward and rarely all-or-nothing allowing for graceful failure is one of the ways we maintain a positive working environment for red team and assure that weeks or months worth of work rarely goes to waste because of a single mistake if we can provide some consistent operational data performance measurement is easier if something somewhere was learned by

someone we'll consider that a success but if not as we see on red team there is next week another way to gauge our red team performance is how much mentoring training and professional development happens as long as our red team is continuously learning and passing that knowledge on to other teams and junior team members we will remain high-performing so in service to this our red team does direct training we know that if our partners understand how we do business as a red team they're more capable of communicating their needs helping us meet ours we present on various topics like reverse engineering malware development to large groups we do debriefs directly with analysts who have expressed that one-on-one time with

our red team is important to them because they get the opportunity to ask questions in a safe environment now our team like others in the security organization operates a shadowing program members from other teams are spending a few days working alongside a red team and getting some white board theory on how we work as well a red team shadows other teams giving us the opportunity to see a much better understanding of how the rest of the organization as a whole functions this process allows us to be more empathetic about how we do our work and how we can get the best value for every hour of every day that we put into it so it sounds kind of

like agile doesn't it weekly Sprint's monthly Sprint's quarterly Sprint's but we've also become a proper software engineering team capable of planning developing delivering stable products we deliver services based on those products we're tightly integrated with other security teams with great working relationships we're capable of measuring what we do and using that to improve week over week we can do really cool things and do them quickly it's been years but we're still at this process we've done it with as few as four engineers we've done it with much better with a team twice that size and since we've become more strategic about what when how we do our operations we're all happier and if drastically reduced the

amount of work required to do quality operations now like I said in the beginning there's no one right to do red team I just hope that our story will inspire you to try new things that for us have driven meaningful change thank you got things to give away somebody named one of the animals that I displayed more time all right got you arctic fox there we go you think