Domain Persistence in Active Directory: Detection, Triage, and Recovery

all right everyone uh thank you so much for being here and thanks so much for the organizers of besides belf fast for inviting me it's a wonderful event and it's an honor to be here speaking to all of you uh can youall in the back hear me all the way back there we're good perfect okay um and I uh landed uh from Virginia yesterday at like 4: uh so if I just stare off into the distance just get out some smelling salts or snap your finger and uh hopefully I'll come back to it um on a lot of caffeine right now uh we have a lot of material to go through um there are going to be some

things that I go a little faster through uh all of these slides which will are kind of meant as a guide will be available on a GitHub uh and I'll find somewhere to share it with all of you um so that you can uh bring it up and use notes if uh uh you have The Misfortune of needing to recover your domain um so a little bit about myself uh me and my cooworker who unfortunately could not make it Josh Prager uh are the researcher uh who came up with the contents of this talk here uh my name is niiko shine like was mentioned originally from Fort Lauderdale Florida uh currently live in uh Virginia uh went to and got my degree

in cyber operations from the US Naval Academy uh did five years in the Navy as an information professional officer primarily working with satellites cryptography RF uh so not at all what I'm about to talk about today uh but uh it's been a fun process conducting This research uh and learning about all this uh also love uh movies live music so any recommendations here to the Belfast area which is a city I've always wanted to visit I'll gladly take those all right so purpose of this presentation when I talk about domain recovery uh inevitably there will probably be someone in the back yelling just rotate the krbtgt twice right because that's the the main guidance that we're aware of uh when it

comes to what do we need to do to ensure that the adversary is kicked out of our domain so the purpose of this conversation uh and I'll bring this back to a client story uh is we had a client that brought in Spector Ops to do some forensics uh and said okay you know we think we have adversary activity in our domain Going Back 2 years we think we've gotten all of it we think we've cleaned everything up and we're fully recovered uh before we bring this to our board we'd like you to come and certify that so we came in and uh honestly it seemed like someone who was surprised they got entrance it was a lot of shotty uh

tradecraft literally you know like malware uncore shell uh type stuff just all over uh some of the share drives that they had um and we found adversary activity going back four years not the two years like they initially uh thought now when we're talking about domain recovery and why that might be an issue um who there probably aren't a lot of people here that could afford to just entirely take a domain offline and rebuild a new one uh with no cost to operations right A lot of the time where we see incomplete or shoddy domain recovery it's because leadership has a a business problem they're trying to solve right we can't take take a week off to completely

rebuild everything because that's losing US money uh so what we want to do here today is say okay so what can we do in an actual Enterprise uh environment to make sure that that domain is recovered and that we root out all of that adversary activity and what we found uh because we conducted interviews which with a bunch of our clients to try to to come up with some sort of Baseline for This research uh is that a lot of organizations didn't really have playbooks didn't really have audit guidance didn't really have any standardization to say okay adversary evicted from Dom we can certify this domain is good to go so we're talking about domain persistence uh we're going

to talk about a bunch of different techniques first right and these techniques could be credential theft methods they could be authentication functionality abuses they could be endpoint management abuses um and these techniques have a few different common denominators so evidence of domain persistence is evidence of a larger attack path right a lot of these will require domain admin in order to carry out the attack if someone got domain admin in your domain you probably have some other attack services or some other attack chains that are going to cause an issue for you right so it's never just okay we saw DC sync well you know it would be lovely for red teams if you could just DC sync uh from the get-go

that's not the case right you you have to build part of a chain part of a larger attack path um techniques represent right that tier zero access that the adversary is able to get so we're going to talk about a few different highl techniques that we often see when we're talking about uh adversary domain persistence and you can see again I won't go through all these just in the interest of time but these are the discussion points for today and some of the techniques we're going to touch on before we move into triage and recovery and auditing and kind of the defensive side of it are going to be credential theft entity s.d access DC

sync golden and diamond tickets adcs certificate the theft and sscm site takeover right and for each of these we're going to do a brief overview um cuz this talk is kind of meant uh to start in a general place and go into the technical and specific so it'll start out basic but we'll move into specific detection guidelines for each of these techniques and then ultimately how we triage all right so when we're talking about credential theft right which is just as easy as right clicking on uh lsass.exe on task manager creating a dump file and then parsing through those uh there's going to be a few different differences we care about when something is opening a handle to Elsas process uh

some methods are going to try to change the the mimic cats uh flag rights right if you see MIM cats with 1410 or 1010 access flag uh I know in a lot of our client environments that's one of the first detections we set up uh so if you're lazy and don't change those access Flags or or try to uh just use mimic cat's default that will get caught right out of the box and some methods are going to avoid that open process of the ls. exe and they're going to instead look for leaked handles uh they're going to try to create child processes of Elsas or they're going to try to create a snapshot um and you see here the

typical operational flow for that credential theft okay right so what's the difference between a host and a credential theft on a domain controller well when we're talking about different defensive measures we can take on the host you can enable credit guard right which uh isolates that lsass.exe process Microsoft in its documentation strongly discourages you from enabling credit guard on domain controller because if you isolate Elsas on your domain controller you're going to come up with all sorts of issues when it comes with syncing or or um uh getting credentials across your domain um so again uh that's going to be the main difference that uh the domain controller actually is slightly more vulnerable just because those preventative controls

are not enabled and additionally you usually have interactive logons with those tier zero admin accounts all right so this is a brief breakdown of how uh we have spectr Ops usually try to build detections right this is usually the course of a few different weeks of research and study really what you ultimately need to see is we have something we care about like process creation we have an entity relationship right a process creating a process and then we link that to either sysmon uh Windows security event IDs whatever you care about and those are things that we can link to right uh because for for us for example right we do a lot of our

infrastructures automated so whenever we do a red team uh there's never going to be the same ioc there's never going to be the same hash it's not going to be mimik cat.exe right so if you have detections just for those kind of Baseline configurations you're not really getting at the underlying Behavior which is what we're trying to do with these detections so instead of just saying okay you know we're going to monitor for that one specific tool here are a variety of event IDs tied to different behaviors that are all linked to that miter technique um and again this will all be available on GitHub and I know uh we're in a little bit of a

hurry but we will um I'm happy to talk with people afterwards as well as how we arrived at some of these all right so another option right to create a shadow copy of the DC itself using volume Shadow Service uh to try to get that ntds DOD file uh that ntds DOD file is normally hard locked while adcs is using it uh and you know in most domains adcs is always using it because adcs is on by definition um so that copy is going to let you access data like the domain uh uh DP API backup key right and we'll get into why that's important later so manipulation of that ntds.dit file uh generates some form of telemetry what

you can do is set a sacle on that ntds DOD file and you can see the settings required for that sackle there um so if anyone tries to modify uh the div file you're going to get alerts generated uh from the sackle you set on it and again this is not default uh so if you haven't gone in and set that sacle on the ntds file you're not going to generate Telemetry if people try to interact with it um you can also look at if the volume Shadow service is started and begins reading you'll generate Telemetry and that could potentially lead to alerts that lead to behavior for example if VSS is running at like 2 a.m. probably suspicious worth

looking into and again we have the same breakdown here right with the behavior we're looking at the entity relationship linked to that behavior for example process requested handle to a file and then the window security event IDs and the sysmon event IDs and you can build these out with whatever tool you use carbon black crowd strike whatever and Link those to the underlying Behavior All right so DC sync uh is one we're probably all very familiar with it's well known lots of detections blog post event IDs linked to it from Splunk Lantern palel to unit 42 uh Etc um and it Rose in popularity due to kind of the ethics with red teams messing with

ntds.dit because that ntds.dit file again has that domain dpapi backup key and so what red teams were finding is okay you know we now have the one key that cannot be changed in your domain that can decrypt all the information in your domain um so due to ethics reasons a lot of red teams move to DC sync and it's it's more popular in that space and again we just have here some of the the detections uh linked to the underlying Behavior All right so with golden tickets um show of hands who's familiar with like the way the KDC TGT all that interacts when we're talking about keros okay so probably good to do a baseline then sorry give me one second

okay so Carib TGT account Microsoft active directory it's a service account it's going to act as the key distribution center or KDC you'll hear me mention that a few different times uh it's going to be Central to the function of that kose protocol with an ad which is used for Network authentication and that's at that high level that's really what we need to know but that Carib TGT accounts primary primary role is creating tgts or ticket granting tickets so imagine how annoying it would be right if you have a file share in your Enterprise and every time you want to open something on the share point you have to like put in a password right

it'd be very irritating if you had to go to your company services you know Tuesday's meeting and put in a password every time you opened up Tuesday's meeting notes uh keros handles that by doing all the authentication uh behind the scenes which is where uh the Pokemon flavored attacks of like golden diamond the latest one is sapphire ticket I assume the next one will be ruby red or something like that um but the typical operational flow see with that right aary needs that fqdn the for the uh formal domain name the S of the domain uh as of 2021 you also need the account you're trying to impersonate uh that's something Microsoft implemented before you could just kind of blindly create a

Golden Ticket uh which is a way of again interacting with all those services without needing to uh authenticate correctly um okay so we have some different data sources here uh event IDs that we care about and these all have to be enabled they're not out of the box um 4768 keros authentication request was requested 4769 service ticket was requested uh so what we see with golden tickets because if you use a tool like rubius or mimic cats you're forging that ticket entirely yourself um you're going to see a lot of service ticket requests right access to a service without the corresponding 4768 because that ticket wasn't requested because you used rubius to do it um and if you're familiar with bpmn

diagram notation uh it's a way of that businesses model um their efficient flows the famous example is you at McDonald's made a bpmn diagram in the 1960s to say how could we be more efficient um I just finished my masters a few weeks ago and this played heavily in that and I thought okay we can take this process um and you'll notice this doesn't look like the previous charts I showed you again because there's so many different things that play you can't really link specific behaviors to specific events so this is just a way of documenting the attack flow for a Golden Ticket uh and we're going to push through again a little uh just in the

interest of time but this will all be available on GitHub uh and you can look at them your heart's content then Diamond ticket so instead of creating your own TGT what are you doing you're modifying a legitimately issued TGT so that way uh you're not having that issue right of the the 4768 not matching up with the 4769 uh but it's a similar process so we have some different event IDs that we care about here and uh what we can do here is if you see that 47 uh six sorry lots of numbers to think about about uh if you think about that 4768 right if you see a 4768 event and no Services were requested uh that's one

tool you can use in Your Arsenal to summize okay someone why would someone request a ticket and then not use it for any Services something to look into and here we have just the the general attack flow for that diamond ticket uh from initial access all the way to uh deciding whether or not the ticket is legitimate okay so some data sources we can use sackless you don't want to set sackless on everything right you'll annoy your sock and lead to alert fatigue and nobody wants that but you can be judicious by placing sackless on objects that matter like that ntds.dit that I mentioned or making sure that cobros has Telemetry enabled so you can incorporate

that in your detection flow you can also look at successful logon events and again just making sure you're tying that to some of the other Telemetry we mentioned all right adcs certificate theft um these Casa they're not really treated same sensitivity as DCS um MIM cats and a lot of tools that for certificates have assert validity of one year so again if you have an adversary with lazy tread craft you can just look through those MIM cats defaults um and then we'll get into some of the underlying behavior in case you get some more enterprising adversaries all right so uh once the private is extracted converted for certificate can be created and signed is ultimately at the end of this I'm going

to push through this technique just to get some make sure we have some time for auditing um when we're talking about ads certificate theft uh some of my co-workers will Schroeder and Lee Christensen uh you can go look at their wi paper has anyone read certified pre-owned yet or interacted with it best friend right there um It's a Wonderful white paper to read it goes through all of the escalations due to certificate theft and the detections associated with them um that'll be linked in the gate Hub and again that's certified pre-owned Wilshire early Christensen uh really really useful and I think invaluable for for socks and defense Defenders trying to make sure they uh have detections for all the

existing escalations that have been discovered and we have here some of the events we care about right uh so I'll break this one down a little for account used key storage provider on key file we have user open key security event 5058 the operation on key file was conducted with KSP right so again each one of these events might not in the in themselves say anything important to us but when they're all linked together they uh tend to point towards adversary Behavior which is what we're trying to highlight right defense and depth anyone that says they can just sell you an outof the-box detection solution that never needs to be updated is lying to you or trying to sell you something

right you need to have a bunch of different layers in your detections so that that the adversary just needs to make one mistake and then bam you can respond accordingly uh secm typ takeover uh who has familiar with this or sharp secm some of the tools out there um yeah so this one's super interesting uh usually with a lot of our red team i k you not it is like 15 minute your domain admin if you can carry outm site takeover um it's something not a lot of people have patched or worked on on it's a super interesting tool and basically what we're doing is uh again apologize for the rush but uh from a coercive perspective uh let's

say there's client a client B and then you have your secm server right if I compromise client a I know the IP address of the site server right so all I need to do is get domain admin on that uh server a um I can say hey give me your ntlm authentication uh that site server then gives me the ntlm authentication to the compromised host uh host a what I can then do is intercept that authentication flow and coers it to another server let's say server B uh and now I'm authenticating to that server that I did not have domain access on uh as secm which has administrative privileges so once the authentication is uh relayed again you can uh add your to

those admin roles and there's a ton of really interesting research one of my co-workers Chris Thompson maintains sharp secm which is an open source tool for secm site takeovers okay so recovering from domain compromise which is really what drove the interest in this talk is there's the the only worse thing than getting your domain compromised is to then walk in and having it compromised again right because you weren't able to successfully root out the adver adversary um so again most organizations have recovery strategies as requirement directed by compliance um I'm unsure of the regulations here but in the United States a lot of those regulations are just people checking the box and uh if you actually look at the content it's a

lot of fluff right for for a lot of the play books or run books that they have uh they often lack quality they often lack valid testing um sometimes domain recovery strategies cherry pick specific items to rotate instead of considering in the entire attack path and how that plays into how your domain was compromised so we're going to look at some things here uh how you determine the scope how you replace or reprovision your domain controllers without needing to take down your domain entirely how to rotate accounts and object Secrets how to rotate your certificates and how to enable additional auditing right uh making sure that you have the Telemetry you need uh to ensure that the adversary

has been pushed out and you can hopefully catch them next time so when we did this research uh my manager initially said to me hey it would be really cool if you could give remed ation guidance for each of the techniques you talked about at the beginning um and in theory right that sounds like a great idea okay I saw DC sync how do I recover from that okay cool I found a Golden Ticket how do I cover from that um but the reality is if I see any of those steps in my environment again what is it a symbol of a larger attack path I would want to go through these steps anyway because how

do I if someone got a Golden Ticket how do I know they didn't use sharp SCM how do I know they didn't use certificate authentication abuse right um so we want to make sure that we go through through those triage and Remediation steps if it's a valid uh adversary Behavior if it's a red team that's different okay so the goal here right is we want to identify accurately what's in scope so that containment can properly reduce the xill uh attack surface so when we think about highly privileged accounts that are both on Prim and Azure uh that would end up putting Azure in scope which is a whole other talk right uh so Microsoft recommends keeping Azure

highly privileged accounts in the cloud own so you don't have this kind of mesh that you then have to worry about um a lot of the times someone pays for a red team right and if that red team gets caught the company says okay well I already paid for this bucket of time hey blue team ignore them like let them operate as normal uh because we paid for it and it's expensive and we want to you know see what else we can get out of it um we personally at least uh encourage clients to try to kick us out and remediate as we go um we use completely automated infrastructure different ioc's hashes again um so reasonably confident

we can find another attack surface but that's one of the issues we have right you're not a lot of times companies aren't able to actually go through the remediation playbooks um because again due to cost constraints or business considerations you want that red team to quote unquote do more work in your domain so when user and service accounts uh trust passwords and certificates uh they can all be rotated the domain dpapi key which I mentioned earlier from that ntds.dit file is the one thing that cannot be rotated as per Microsoft documentation Now That's What Microsoft says I have a coworker that found that you can indeed rotate that domain DP API backup key what does it do it's it's

responsible for encrypting and decrypting everything within the domain so I want everyone to think for a second if I change what is encrypting and decrypting everything within my domain and that fails what did I just do to myself I just ransomware to myself right for free uh and there's no way to get that back so that is why Microsoft recommends do not touch the DP API backup key uh because I would imagine there would be some legal complications if uh if anything bad were to happen um but I do one of my co-workers uh is actually on a research sabatical right now looking into methods of doing so um so it's possible uh but again it's just

up to you and your leadership do you trust X third party tool to not ransomware my entire company um so that's one of the issues we have right and again that's why ntds.dit is such a big issue anything less than creating an entirely d new domain from scratch that's the only way to create an entirely new dpapi backup key again per that Microsoft documentation and if you can't afford to start from scratch and almost no Enterprise can uh here are a few things you can do instead and you see all these steps here again the purpose of this is you can go to GitHub and go in your sandbox or your lab and go through these

steps yourself to test it out um I won't necessarily be reading them line item by line item uh which hopefully is a good thing um each of these methods me is going to have different curveballs that make that process more difficult and again keep in mind that dpapi backup key is really critical when we're talking about this overall and we also care about those flexible single Master operations the fsmo roles also known as the operations Master roles and what we're doing here right when we're rep replacing rep provisioning our domain controllers we're creating uh an existing server that's not a DC right so hopefully it hasn't been targeted or compromised we can verify that we're

applying our patches to that server we're adding adds that new server we're promoting it to DC and then we're valid validating moving all those roles from the old DC to the new one which hopefully has not been compromised and again here are some steps you can do and it's very critical that you verify the health of your new domain controller as you go through these steps and it's very critical throughout any of these processes that you wait for replication to occur um again if you move through a step too quickly and something hasn't replicated across your whole domain you just ran some word your domain uh so you want to make sure that you're running

through these health checks you want to make sure um that that's not going to occur all right so again with user accounts we want to rotate these uh usually they involve one or two compromise domain uh admin accounts and those are Central to the larger attack path so literally within 5 minutes 10 minutes uh this should be your first SLA you've identified the domain persistence on a controller you need to find the compromis tier zero assets and immediately disable their permissions um you know you could have uh in a more dire situation you have to disable all of them which hopefully is not the case but that's something to keep into consideration that's what you want to do first right

take away their immediate access to that DC so they can't pivot or laterally move or laterally move all right um so service accounts those can be temperamental or Legacy when we talk about rotating service accounts um I did an internship for a municipality in the United States before I found my job at spect drops uh did network security there um uh it was for utility and wastewater treatment and I was trying to build up their security program for their modb TCP on like all the controllers and pumps and things like that and linking that to Enterprise detection in their sock and I said okay first things first um where do we log all the TCP mod bus traffic logs oh okay

and that is why I did not stay there um but when we're thinking about service accounts rights a lot of that Legacy equipment and especially if we're talking about companies with large industrial uh components those service accounts can be 20 30 years old and the password could be password 1 123 but if you change it then you're locked out of a certain part of your Enterprise and I see some people nodding so I I know it's a painful point so before you rotate those service accounts work with your service engineering or security engineering department uh they know what needs to exist and where and what could be broken if you potentially do some replication or account rotations

all right so this is again just talking about that collaboration and really this is what this should be uh you don't want your red team your blue team your purple team whatever the case is working in isolation uh you want everyone to uh be sharing information to be collaborating which again is why those run books are so critical if you as an Enterprise don't have domain recovery run books it's something you might want to look into right and you want to look into okay I need the sizo to talk to this person I need the head of the sock to talk to this I need our data Engineers to make sure to find and locate the

accounts that we need to operate our business and this should all be built into those run books that we worried about um luckily enough uh when we're talking about rotating service accounts like the krbtgt Microsoft has a lot of valuable command lits and Powershell scripts on their website official Microsoft Powershell scripts you can go through they'll be again linked in our our um GitHub and this one uses new T carab TGT keys. PS1 uh which is the Powershell script that Microsoft provided in order to rotate those terab TGT accounts and again wait until replication is complete uh don't do this in like 10 minutes and then come blame me when you've ran more yourself you

want to make sure that replication has spread and that everything has occurred quickly if it takes 24 to 48 hours to rotate some of these that's what it is um but again you're going through these steps to hopefully keep your Enterprise up and it does you no good if nobody can access the Enterprise uh so that script that I mentioned is going to change the krbtgt account password and validate that that replication was successful all right so uh uh machine accounts again it's not going to remediate coercion and relay uh there's a hot fix which I have here on the bottom right uh that you can look at at a later date and that'll help you uh deal with NTM relay

issues okay I've two minutes so we're going to briefly touch on this so enforced with multiple domains Microsoft uh and its documentation uses a shared secure password and converts it into cryptographic key and what that key means is uh an interr trust key right so again if you're the window domain was compromised and that inm trust key was compromised that means the domain it was linked to is also compromised uh so that's something else that we in our research we found you need to research and deal with okay uh with Casa you want to initiate a revocation process you want to update the revocation labels and you want to republish your certificate revocation list as you go through this

process okay so for additional auditing every time a domain recovers from domain compromise leadership asks how can we stop this from happening again uh is you can't right there are too many attack surfaces Too Many Humans where with humans being the the the weakest chain um so what we can do is turn on additional auditing go through those sackless making sure we're getting event IDs associated with specific attacks uh to try to reduce adversary dwell time because that's ultimately what we're trying to do they only have to mess up once hopefully for your detection deam to be able to operate and act appropriately all right um how do we know which new logs to ingest uh and tools like blood hound you

can have an alert uh or logic to say hey this account is kerb roastable or you can have hey someone downgraded to rc4 encryption right some different tools you can use go back to those events I mentioned in those entity relationship diagrams and break those down to try to see okay what is the atomic Behavior targeted by the uh tool or technique that I'm worried about uh so in conclusion uh a lot of organizations don't have custom detections for this right they don't have run books for domain persistent behavior um they're important because they're part of that larger attack path they are never in isolation and when we're talking about recovery we're trying to reduce that adversary dwell

time and have playbooks to help you recover from those scenarios successfully uh here are my co-workers uh that helped me with some of the sharp SCCM Alex Sue is the one that is working on the dpapi backup key rotation um and a lot of my other co-workers who were very kind uh with their time and helping me kind of work through all these processes and that's it I I think exactly on time right okay thank

you thank you do we have any time for questions okay perfect uh so we do have time for a few questions if anyone has any I can go back to stuff I already showed as well because then we kind of rush through some [Music] things seriously

um uh sadly on the client perspective not really um because it's not for whatever reason something that blue teams or socks cuz socks are already overburdened as it is um and I've there out of the 20 clients um I've worked with since I've joined spectr Ops only one is what I would call mature from a detection standpoint and they're a fortune 20 company so like it just takes a lot of resources to do that um but um yonas and Andy Robbins the two architects of Blood Hound uh publish new escalations all the time um so uh spectr Ops we have a medium page with all their research so new escalations are there as they're put out um and sadly there're

only five of us on the detection side of spect drops but we try to go through and match it with detections as they come out with the escalations yep yes I'm very sorry I can't hear

you yeah so um the question was if you have that interr trust key is compromised like do you have to rotate everything in both domains twice if you find that evidence abser Behavior then yes and again this is just I'm trying to talk from like a realistic perspective the the good theory answer would be you know rotate it all um if you don't find adversary behavior on there you can kind of keep a lid on that while you try to deal with the initial domain that domain trust key though would need to be rotated itself how much more time do we have oh perfect any other questions now anyone have any uh slides you want me to go back to I know we

pushed through some of the diagrams and stuff pretty quickly all right yeah absolutely yeah I'm going to put um the GitHub has all these slides as well as some of the test scripts and uh links to all the research we have and the links to all that Microsoft uh the official command LS and scripts that we talked about um so I will pass that on so that can get sent to all of you and again you can kind of go through those steps on those slides and sandbox in your lab environment uh and test it out for yourself um and yeah and I'll be around so feel free to come up um uh it's a it's a a big topic that I think

is kind of underserved by the detection Community uh so I'd love to hear any and all feedback thank you again