BSidesCharm - 2018 - Exercise Your SOC: How to run an effective SOC response simulation

okay good afternoon everyone my name is brian henderski i work for u.s citizenship and immigration services as one of their senior cyber security engineers and here i'm going to talk about how to exercise your sock this is a concept we came up with internally about two years ago we were trying to find out after a major ir had happened on our network basically how we're going to go back and validate that in production but not let our sock know at the same time when that was occurring so some disclaimers because i am a federal government employee these are my when i state a personal view it's my personal view it's not an opinion of the federal government dhs

uscis or anywhere above uh mention of any product name in this talk uh it's not a government endorsement any questions raise your hand i i have no problem with questions midstream it's about my work we're the largest world uh immigration in the world so we have 8.5 million applicants that we process 1 million permanent records 730 new citizens and of those are military 7 200. we're fee funded so we process over four billion dollars in fees a year and over seventeen thousand employees in over two hundred locations and overseas and uh department of state's consulates and we are leading federal agency in the cloud besides dod i see about me i specialize in appsec devsec

container sec hardening and a lot of security design so i assess system for weaknesses and design i build the risk mitigations and help build the templates and secure by default and highly embrace infrastructures code and rugged devops for repeatable configuration also ipt component representative for dhs cloud migration which are going through right now devsecops which is uh slowly spreading throughout the government sectors and the cyber threat intel groups that they're standing up now in addition i advise several other federal agencies uh with the two years of lessons learned that have had at uscis on the fed side that are not classified and using cloud agile and devops methodologies to perform and irony was talking with irs a week

before their issue about how to do devsecops and configuration control prior life i was u.s air force civilian with dod cyber crime center i first started there doing work first development the dc3 digital forensics challenge i did the last three years of execution for the program for that i was at the seat for the u.s cyber challenge in addition i was doing a dibnet so that was all the defense industrial-based intrusions being reported in on a voluntary program before it went over the deforest the mandatory doing tool assessments later became a program manager and then security assessment the one big thing that i did at dc3 is participated in cnci 5 slash essa where we basically took the federal

cyber centers nsa ied u.s cyber command fbi cyber department of energy and dc3 and we connected them together with the first versions of sticks and taxi with mitre prior life before that um did a lot of it dev and ops i did commercial health care consulting i was a former.com boom and bus developer and also in academia doing help desk so i worked my way up personal philosophies trust but verify i know reaganomics and all that but this is kind of something that i like that hey if you have a question about something you're questioning it you should actually test it and see if it does what it's supposed to art of war attack is the secret of

defense the defense is playing planning of the attack and my latest one is burn down your technical debt before it burns you because i see it happen all the time where people will say oh it's acceptable risk and then you'll come back years later and find out no one really knew what that risk was really going to be and guess what that's what your ir is today our security challenges since we're a little bit leading edge we have a lot of shiny new things i get a lot of things that are not 1.0 that developers want to use i don't have much love for apache foundation because security is usually sold separately incident response when you're doing

agile devops so when you have a pipeline that's automatically pushing to production four times a day that can kind of get a little interesting when you have incident response because your machines will change out from under you so you better hope your logs are good and then you are the cyber cyber this cyber that cyber so as security we're usually usually the last to know but the first to respond uh acronym soup i live dod life so i have a lot of acronyms in my head sometimes i'll use them so i'm just defining a few of them in case people don't know um apt advanced persistent thread ak attacker ir for incident response in this context

is typically sock ttps tools techniques and procedures and stock security operations center which is typically your ir response team in some larger organizations but some organizations may not have that so why bother doing a sock exercise you want to validate your implemented ttps for your stock operate as expected you want to test the effectiveness of the attack of course in a controlled execution and yes you do want to test it in production you don't want to touch the staging you want to test it in dev because those environments can act a little bit differently and you also have the additional environment variables of how that production environment may operate and find your actual defense thresholds

a lot of the policies and a lot of the systems are configured saying oh it's going to trigger this alert at this time and when you do these type of exercises you find out well all these other environment conditionals might push that threshold a little bit back what's the end game uh air force is all about bluff so this is kind of the bluff about that avoid becoming whatever major news article this is about proactive training so you can consider consider it a part of your audit you can also consider part of your training exercises with your soccer accreditations or your general accreditation programs for especially for the attack patterns learning together so when i first came

to usas i was managing both the apsec and also red team so i had a unique ability to say okay red team i want you to go after this here's the source code here's their configuration stop operating in a black box let's be effective and also determining where to focus everyone has very highly limited resources and where to reduce those risk of future compromise or set up additional alerting remediation tactics now building your team so the team rolls red leader uh they have the com the official comes all the cisso and exercise execution they lead the team through the exercise build run and closeout it is a process a gold leader they're the ones that

monitor your chatter um on the ir detection um sometimes you can have them watch sock but then you got to watch it because some socks watching you and also recording the outcome of the events when you execute and then your execution team typically you want to keep a small and tight team uh one to two red teamers specializing the particular attack vector that you're going to execute and one or two blue teamers that specialize in that particular defense so they know its configuration and possible unexpected behaviors that may occur when you're going through the exercise so this is the critical concept it's you're testing a hypothesis you're not destroying your targets you're doing objective testing of the

ttps you're assuming professionalism of the targets involved it's not trash talking them and if you can use dummy data that's be and modify benign attacks that way you can simulate an attack but you're not doing any real damage to production and let the results show the actual true security posture don't make assumptions don't be subjective this is about being objective being used in scientific process and the purpose is to be detected that's the whole purpose of you doing this exercise so you want to run low and slow at first below your thresholds to kind of get a feel if you're going to pop them increase the noise over time on a dedicated action plan until you to

expected detection and plan for go beyond expected detection because a lot of times when i run these exercises you have to pump up to 11 or even 12. another critical the ciso owns the risk so when things go south they're the ones are going to have to sit in front of the business ops management and the cio to explain what occurred they give all the official green light yellow light and red light for the exercise run and they pick the date when it executes in time because they typically have a better idea of what's the yin and yang of the business operations at the time especially at the i.t ops level i kind of touched on this a little bit so

you want to limit the knowledge of the exercise life cycle keep your need to know at least individuals space out your execution like most apts you don't want to do recon immediately and then start attacking you want to space it out it kind of spaces it out in your logs your seams and you and your tools and then use a trusted comms channel so typically we like to use slack we do a private channel and we'll do a war room so we'll start recording all of our recon our development do the actual run time there and then use that as part of the report recreation and guess what a date and time stamps everything so it makes a lot

easier on the reporting end from a timeline analysis anybody have any questions do use slack for right now of course there was just an article in mexico that slack might be going away uh major steps so the process of developing the exercise you want to do your target selection determine your expected outcomes you want to brief your system to sign off your exercise day and then a lessons learned debrief so through your target selection i'm going to state a hypothesis to form a conclusion when i run attack a against asset y then these things should happen defense d should trigger process p should happen response r should occur recon your targets observable so i'm very big about sticks and cybox so

mapping to those for your ops and your risk leverage your historical logs for events at the timeline analysis like if you had a prior ir determine the target's i.t and business ops lows and high so you're not interfering with critical business operations research target dependencies and outside ttps are dependent upon it now like i brought up before about the irs they didn't document all their system interdependencies so when their caching mechanism and the irs mainframe locked up it locked up 10 other systems right behind it and i'm a very big proponent about crawl walking approach i had a previous six sigma background when i was in healthcare for their risk management approach so use a laser-like focus

by approaching existing data target mission critical processes first you want to target the four percent to give you the 50 percent most pain and honor the process to build the trust you have to use that record of success for your exercises starting out small to build into bigger projects and i'll go into that in a minute once you do this you can start reusing some of the exercises to help reinforce that you've actually implemented some of your lessons learned or tweaked them a little bit to push it a little farther so for a crawl to focus on a simple rule threshold or a past ir example if i run exercise actions below or at or beyond a learning behavior let's say

like a dlp what should happen walk coordinate with others outside your immediate security team example explicit permission we had we had a we had an inside project called wild turkey which we took a person that was in one of the processing centers we asked her that we could borrow her user account and we asked her to stay home and we used her user account and her boss knew she was staying home and we started escalating her privileges slowly over time and see who would notice and did they file the proper reporting procedure to call her they did to disable her account not as fast and then report it to the syso run complex exercises that can be

multi-day so typical multi-week exercise where you pivot on one machine and establish a foothold and then you start using mizzy cats or something like that start grabbing creds off the wire and see where else you can go and slowly pivot and pivot until you get found and increase the noise so here are some examples of how to do a detect uh when will the alert insert attack vector occurs in this system at this time has this tool been configured to operational environment beyond its defaults what you'll find is a lot of places we'll just use the defaults throw it in there and go will this record an audit log of course i have a prior dod

auditor background i love audit logs well is this event actually recorded as an observable so i can actually mine it out later or build alerts around it uh on the respond did it respond to the attack vector some tools say they do things guess what when you put them in the real world sometimes they don't do very well or if they're under different environment variables they might not operate as expected does your stock report capture those observables because that's capturing the skill sets of your stock and depending what analyst worked on it or analyst and will the tool from the socks perspective log what they did so you have so you can watch their activities

and then all your escalations did you follow your ir uh are you meeting ir response times you have in your incident response plan for that particular system or for the enterprise and uh one of my favorites is uh when a critical person the ir plan is out like around the holidays what happens does everything start falling apart uh determining your expected outcome so you got to first develop your timeline of your events of major actions uh quantify the potential risk to his execution because this is for the cso and he's got to sell it to the cio and establishing your measurements on those actions so do execute the attack the known defense thresholds but build your actions increase over

time and exceed those known thresholds and plan for execution failures and things to fail things will fail don't execute an attack you've never tried before you'd be surprised and always assume that and never assume the attack's going to be successful you're going to have unsuccessful attacks sometimes it's just a crank in the wire just because of what traffic's going on risk management now kicks in so identify the critical operations at the peak usage to whatever target you're looking at look at the execution during other incidents you don't want to do that because then socks are going to be tied up you don't increase their pain too much you don't want to break an unrecoverable system there's a lot of those around in

federal government and there's a lot of those in commercial industry where someone set it up that system admins long gone and nobody knows how to work that box anymore and you're always going to run into some unknown interdependencies because through workforce attrition uh then calculating that probability of that event so let's say in action threshold t triggers for observable o and stock tool s1 so the probability is high we know that it's configured that way the impact's low if it bypasses that and we know that because we did recon and we know it's manufactured defaults

so how to do the measurements results of the action so again stixx observables which was previously cybox you should be able to map to those and you can map to those then you can start talking about a persistent thread a little bit better and also relating it back to iocs and other observables uh you should hopefully have security events and your security tooling for both your actions and your reactions and you should be able to aquarium in your seam hopefully and hope that you and it's also verifying those audit logs are actually getting pumped to your seam because you'll find some systems aren't and be aware your sock may be watching you because they have

access to the same systems and i've had that a couple times so use objective measurements you only use subjective you want stuff that's going to do basic uh statistics time between actual versus re actual versus uh planned time to take to execute an action start and finish frequency of an action and successor failure of an action you want to prepare for other findings you may run into unusual output that wasn't expected discovering other possible weaknesses in defense that happens that happens from time to time and ensuring documented post-disclosure of the exercise when you do find those kind of things because they become other exercises or they have to go into security plans and further research

all right now you got all your packages together you got everything developed now you gotta go in front of the cisso so you need to be clear on the purpose of the exercise you gotta have to speak management speed so for this one i'm giving an example you're evaluating your dlp sensors at location x so it's a limiting scope on our network that detect pii exploration to policy y so you're testing your policy to say hey this dlp is configured in a certain way in this location to operate to these certain parameters and explain why this is important and then discuss have a discussion about that action plan because there will be questions the ciso has to be able to understand

and communicate that up to leadership when things especially go wrong sometimes they do confirm the accuracy it will also help you he or she will help confirm the accuracy of your assumptions because they know the business a little bit better in the i t ops and the leadership chains and based on their feedback adjust and one other here is establish your parameters when you're going to start and stop your exercise you got to consider the business impacts i t ops impacts and any actions that exceed the expected thresholds of course the joke is have a safe word

and then establish everything in writing once you've had these discussions this is your get out of jail free card as most pen testers know you want a written agreement that signs off the risk for you to be able to do that require the cso the physically you're digitally signed print it out especially doing physical testing because in case you get stopped by security and start getting called off then you can break out your jail out of free cards this is part of this exercise called my cisco and explicitly obtain exercise days with the cisso expected days expect the data change for the ops rhythms it can happen and establish a method of comms prior to

go time like we did with a dedicated war room so the system would literally drop in there say okay we're going to execute within two hours then the fun of exercise day so you got your war channel going you have your team channel ready everyone's ready and the cisco the cisso gives the red red leader green light go we learned after some time let the system sit in channel um one they'll enjoy it and two if there's any other things that you don't know about that's going on in the network um they at least know if it's you versus somebody else um we've had incidents happen while we're running an incident reporting the action of progress because

they're going to want to stay in tune that helps if there's conflict and immediate report any unusual observables that you're experiencing when you're doing the exercise one that goes into your reporting and two that goes into further research if it is another incident and explicitly report your ttp thresholds when they're exceeded because that's the critical point of this exercise is you're saying you should have detected here but you're really not detecting yet before you start pumping that volume up to 11. and when you're recording those objectives make sure you're getting screen captures photos of event outputs changes in the timetable for event execution individuals responding to event and educate yeah event execution and then document

then at the end of your exercise confirm with your ciso on the exercise closure the system may want to extend a little bit further especially if you're not disclo if you haven't hit your threshold rate as expected and have had that a good one of that is we're actually sitting in another part of the dhs network and we actually had our red team deployed we wanted to test the policy enforcement points between our component versus another component we set the red team over there we started with a simple nmap let's start port hopping let's start ip hopping let's see if we flag okay nothing all right let's increase the rate okay nothing all right let's go

start going sequential nothing okay let's hit manufacturer defaults okay see it popping on logs but nothing's popping let's start turning that up a little bit okay we're not getting any calls from any of the socks yet um call up the cisso all right sir it's already been four hours executing this exercise can you can we go full throttle so max out and map let it drop bits and the phone rings in five minutes because they weren't watching that particular connection between those two components that's actually one that won me a dhs engineer of the year award and the dhs cisso had to sign off on three poems for the entire network they've fixed it since then

uh host a post-op call with the team in the cisso it's very valuable because then the ciso is absorbing some of that information in case he gets questions from his bosses so collect your thoughts in the runtime results that way your team can start sharing things back and forth document the lessons learned then during your runtime because once you're done off that exercise you start going home and relaxing you're going to start forgetting things discuss anything that was really unexpected because it may be something else that's running around in your network or it might just be a ghost in the shell of the network and that initial report so you create the briefing i know where every

government loads powerpoint to death so your audience yeah your exercise team and your ciso any persons that were impacted by the exercise testing because which guess what you're going to give the briefing to them to let them know what you did and they can start giving you feedback of what they experience on the other side and assume it's the exact level in case the deck does move up make sure to provide the purpose of the exercise show the execution timeline from expected to actual results and keep a tab on live briefings for the q a and capture those q a questions that's kind of a lessons learned thing from doing agile and the briefing be interactive maintain

a blameless environment keep just to the facts typically as you know from red team sometimes sometimes the people that got attacked are a little hurt solicit feedback from them force them to give them feedback things that they saw things they found that would be unusual because it may help in your alerting and tweaking what you need to do from your monitoring standpoint and then propose possible solution solutions to mitigate or correct those discussion points because guess what those are going to be brought up later for continuous monitoring so provide the actionable goals so leverage the input from the targets and quantify that with the exercise and then start building that into performance either at the program level the system level or

start building in your continuous monitoring or your your stock dashboards and your typical playbooks

questions yes

because she purposely told her boss that she was she didn't tell her but she went home so she was supposed to be at work that day called out sick her boss didn't get the message on purpose but we had coordinated above her boss to let her know that what the exercise was doing so that created a little bit of confusion

yes

right

yes so we have taken samples of some of their malware through our red team neutered it a little bit so it didn't go out and do some certain things like route at the dev slash null but still let it beacon out and those kind of things but i'm lucky because i have a really good red team and um the funny part is because it's a small community i actually knew them all from the mid-atlantic ccdc so it was kind of a reunion when i came into this employer and i saw them sitting there i'm like oh we're gonna have a good time [Laughter]

yes uh so our a so where i sit in the agency's little unique so i sit in a cyber defense branch where we own both blue team and red team but our sock is actually in enterprise i.t operations so they're kind of separated uh so within our own team we can kind of coordinate that but when we start needing other resources we have to go and coordinate with their with the business or their boss and work that out and keep them under a kind of like an nda but it's not really an nda for that exercise what will happen is um usually that business leader buy-in especially when you're doing the walk in the run

and the cisco will make that agreement themselves and talk at the executive level and then they'll kind of work out when that go time is so it could be i could be told a week ahead okay we're going to go on wednesday no borrowing nothing else goes on then wednesday morning comes up was like all right status check in and we're still okay you're still on two hours and when that two hours comes up then ciso says go it's just making sure we're not screwing with prod and we're screwing with the business again remember we're processing fees so when we're down for an hour we're not processing a million bucks in fees

yes it does for some of them like the end map um some of them we're happy because we're detected really early it's like oh we exceeded your expectations this is good and but sometimes you'll find some things in the process that didn't occur because people are escalating above their chain

yes in our environment

than i would

so the advantage that i have is that i have direct access to the risk management branch which means i have access to all the system packages which means i have access to the list of assets so based on that list of assets i can make a determination okay this is what manufacturer default is for this through a little research or just calling the vendor up and then taking whatever word that person says and kind of adjusting based on knowing and talking around there's a lot of history with some of the ops people and especially on the security side you have to be partnered with the ops people in order to get your job done so through some of those partnerships

you can kind of quietly ask around about certain things and make sure you pace it out that it's not related to the exercise over time but if you're coming in blind like if you're coming let's say you're doing consulting and say you're not doing fed gov you're coming as a consultant you're trying to execute these things i mean a lot of this you're not going to find out unless you're going through system documentation or the customer is literally handing you configs you're going to find out the hard way or the customer could be like well this is what our policy says test it to that policy and see what happens but some of it's educated guesswork

because sometimes you don't have all the answers for where it's at that'll help answer your question or no uh yeah i know it's it's

right exactly again risk acceptance anybody else have any questions yes

yeah we repeat it through process the problem is each one is kind of a little bit tailored to what we're doing in our networks [Music] we also have the advantage of being remote and we have 200 locations we can pop into at any time so we have a little bit of flexibility there and we also can kind of hide in some of the larger headquarters traffic so that when we're coming through those kind of lines they don't easily see us which adversary has the same methods um be a lot easier to execute on smaller companies but not the largers because yeah i'm thinking back to my healthcare experience where there's only a few of us and yeah

yeah right the only way you're going to be able to do that as you start establishing sensors out in the field and then start activating them when you need to do these type of exercises or you just literally take over an rdp session on a box you've done that as well sometimes on some sites

yes uh we try to do at least one a month uh yeah yeah plus our annual pen test on top of that too we have about uh oh it's public there's 110 fisma systems we have and of those that are fips 199 moderate and high high value assets a good 30 40.

so again it goes back to using existing data so either we can repeat a prior attack to see how well we would respond to that like um wannacry um or you can do particular irs you've had in the past like amazon spill and then try to repeat that against another team and see what happens so there's always plenty of material it's a matter of execution time and sometimes what we'll do is um we might have like a week-long session where we'll pump we'll pump out about three or four of these different scenarios and put them up to the table to for system selection yes oh sorry i didn't see okay [Laughter]

uh my co-worker is actually doing tabletop now because when i came on i was supposed to be only appsec and of course i got other duties as assigned so he's handling a lot of the red team sock exercises now and i'm helping him at the higher level of the execution because he's learning the ropes which is why this actually is good timing because he's he gave input into this deck and feedback and i had presented this uh information to the dhs cisso council a couple times but with a lot of other attribution it's been removed out of this deck good old public affairs

yes

it's all randomized anybody else have a question cool thank you guys okay