Came for the Ransomware, Stayed for the APT - John Dwyer

all right what's up nerds okay thank you first of all for coming out and listen to me talk to you for the next 40 minutes or go i don't know if you guys saw justin's talk before this but justin you are the most casually hilarious person i've ever met my life and i don't know if your memes got enough credit so allow me to say it for everyone else that's thinking it so uh i'm going to try to jump right into this because i've been described as verbose and i don't want to take up the rest of your day so my name is john dwyer and i'm the head of research for ibm xfors if you're not familiar with xforce we are the consulting arm of ibm security so we offer defensive offensive and threat intelligence services one of those defensive services is incident response which is the team that i came up through uh over the years and now my job is the head of research is kind of two two areas one of it's kind of like a devopsy role where i'm overseeing technol technology developments to make sure all those services are operating as efficiently as effectively as possible the other side of it is to kind of highlight all the interesting findings in things that we've had over all these different engagements and make sure we're sharing those with the community i try to do a good job of releasing like kind of tactical things on my social would be like detection opportunities or interesting findings on malware just because we kind of want to be you know operating appropriately in the community try to burn as much of the ttps as possible my background in a word is chaotic i would say so i i started out my professional career as a laborer went through help desk actually i don't know dugus are you out there right now he might have my first boss ever who gave me a chance is actually in the audience today i ran i ran into him um but i've done a whole bunch of different things over and i've gone up through it and kind of pivoted into security and then into research and uh at first it was kind of like am i going to be too much of a generalist and then over the years it's kind of turned into like a slumdog millionaire situation where things would pop up like when proxy logon and proxy shell happened and i was thinking back when i used to be an exchange admin like oh i get that i understand how kaz's work and i understand how they interface with iis so i've been all over the place i have up there that my superpower is okay with being dumb mostly for two reasons number one is that i never think i should take myself too seriously and you know it's one of those if you can't laugh at yourself who can you laugh at situation number two i try to bring this up as much as i can is that i wouldn't call myself the most gifted person ever uh and when i was in third grade my third grade teacher told my mom that i was probably going to struggle in life because i had such a hard time learning so that's always kind of stuck with me and it always seemed like i had to work twice as hard to accomplish anything in my life and that persisted throughout my career uh great story those of you who know me it's like you're going to get a lot of tangential stories as we go through this um the first time i got like a sis admin job i was like i'm making it like i'm doing the thing and i was really proud of myself guys i went to that first team meeting and it was like these dudes are talking a completely different language i had no idea what anyone's saying it so i'm like feverishly taking notes about stuff that i'm gonna have to look up after work because i don't understand and it was embarrassing and it was like disheartening to go through that process and that persisted as i went through every single evolution of my career and then i as i got older i started to realize that if i'm continually putting myself in the position where i'm the dumbest person in the room that means i'm choosing adversity i'm choosing to grow rather than be complacent and i look back on that now with hindsight being very old at this point and i'm very grateful for that time but if you go out on twitter or linkedin like it just seems like everyone is crushing it like everyone's putting out amazing research and all these great tools and it's really hard if you're just starting out to not slide into those negative thoughts about yourself or about your career so i i like to bring this up anytime i have a group of people just in case there's someone out there who's just starting out or or they have that imposter syndrome to say that you know you're not alone we all kind of feel that way and if you put the work in you can accomplish some great things with that said let me get let me tell you about the one of the coolest irs that i've ever worked like i said it came up through the ir team and like if there's any responders in the crowd everyone knows shiz goes down on a friday right so people call on the friday a client calls in like many other ir firms we offer a hotline they call in to declare an incident so they say in one of our one of our domains we've been hit by a massive ransomware attack and they're pretty large retail organization nationwide three main areas of their business but they got shops all over the place and at this point i mean this was the golden age of ransomware it was like every three days we were getting one of these calls so it was like there's another one here's another one so they call and they said about 200 systems they're offline they call in they you know it's actually kind of cool the av vendor they're not responders but they did some analysis found the domain admin account and some an ip address they blocked it and they called in they're like hey we just want to run past this is it cool and we're like no dog this ain't cool like there's a lot that we need to scope right so part of being a good responder if anyone's trying to get into the industry is you know sometimes you got to save the client from themselves right so we talk about you can't move forward right now this hasn't obviously been scoped as well as it needs to be we need to increase our visibility the only tooling that they had right now was av so we go through the process of the ir and like i said we've been doing these a lot so they said all of our files have a dot ryk immediately we know that's rio grande where and at the time that was the hottest one so we got that all the time also at that time we understood that the emote trickbot empire ps exec ransomware we knew that that was a thing and we did that legitimately i think i ran the stats on it was every three days for seven months that we responded to more than what we considered as a major ransomware incident was more than 200 systems so to put that like this was happening a lot one of the first things pro tip when you do an ir always ask for a security telemetry always ask for it there's gold in those av logs and while we're deploying our tech we're collecting that data that all takes time but you can usually find something that's going to point you in the right direction in those av logs this was yet another example in fact a couple weeks ago we put out a research paper analyzing why are ransomware attacks happening faster than they did before which was a cool topic it took a lot of time but one of the more interesting findings in that that i think got glossed over by the media was year over year there is more evidence in security telemetry now before ransomware was deployed than there was in 2019 the start of what we would call the golden age so that means that people are a investing in detection tech and b the detection tech is actually working better than it used to and there's usually something in there that could prevent you from having a crisis so make sure you're collecting that make sure someone's looking at it first of all and then if you're if you're an ir person make sure you collect that the other point i want to point out here is you can see that the older av alerts responders are going to know what i'm talking about here so older av alerts show someone trying to dump lsas show cobalt strike show them interpreter now what is weird about our timeline here and and bear with me here i'm not going to try to sell you anything but why i think every organization should be interfacing with an ir team in some regard an ir consulting team if you think about this ir firms it doesn't have to be again it doesn't have to be export it's great if it is but if it's not no big deal but ir firms are one of the only disciplines in cyber security where you can can continually observe an adversary be successful right there's a lot of smart people and the mdr mssp avs all these they're putting out fantastic work but it's in their best interest to start to stop the adversary before they're able to complete their objective with ir especially with ransomware the detection point is typically the ransomware itself right so that means the adversary is able to go from initial access to impact and ir teams are able to reconstruct that story and they're in a unique position to say what is anomalous and what is not from an adversary operations point of view and we talk about strategically how we need to operate now and in security is moving out of indicator driven detection strategies and moving into how we know adversaries obtain their goals and objectives so this bottom thing here is incredibly weird because we know from doing all of these rare ransomware investigations that trick bot should be the first piece of evidence it should not be some other c2 framework that is generally associated with humor human operated activity so like they say back home that's wicked weird we're continually bugging the client about this again being an ir person one of the best things you can do is protect the client from themselves but understandably they want to get operations up as soon as possible we know how to do the trickbot empire ps exec we know to do that investigation we find all the domain admin accounts that were compromised find all the lateral movement find where they stole data buttoned up three days we're still asking about this system because it's only one system that's hanging out there that has this av alert that is the only thing there's two av alerts out of a sea of mess that we're bothering the client about but they want to go ahead and go with recovery so literally typing the report out saying with a big section that says dude you might be pop still we're not going to sign off as this being a remediated incident you really should look at this the it admin texts me he's like oh i got that system dude do you want an image do you want edr deployed or do you want us to run your tools on it i say yes do it all right and immediately what we find out is that the same system that had meterpreter and cobalt actually also had the earliest evidence of trickbot it also shows that trickbot was introduced through a meterpreter session that's odd right we know how trickbot goes right it's spam caming or downloaded through email emote that's incredibly awkward and weird to see that data point the other thing that we see is that the user account that associated with those alerts is from a whole other domain it's from the hq dom well hq none of these domain names are real but you know what i mean so it's from the the root so the hq domain was the the forced root of this domain and it was 37 days before ransomware happened so we have to expand scope right so we call hq and we say you know houston we have a problem kind of deal like we need to start ex uh expanding our visibility and they're like well we didn't get ransomware so like we don't have a problem like dawg you got a problem right because this is this is abnormal but you know they also didn't get ransomware which was weird at the same time so we're moving forward expanding scope trust the process collect the data look at the data what do we find out is that there's cobalt strike within that infrastructure 96 days before ransomware now trick bought to react on our research is the longest life span of all ransomware attacks we've seen them prese persist up to three months at the max 96 days is a behavioral anomaly based on how we know these attacks happen so 96 days that they're in this environment they have cobalt strike then we find out that that account that was used to pivot to west is an enterprise admin weird right and then we ask even more that enterprise admin account the security and i t teams have no idea who's associated with assumption the adversary created it for us you know a few guys aren't windows people and and if you get enterprise admin you effectively own the entire forest right so you've owned the whole thing but you didn't deploy ransomware in hq right hq is the largest domain within the entire environment it's got the most critical systems if you wanted to create leverage why didn't you deploy ransomware within the hq domain incredibly weird right none of this makes sense multi-domain ransomware attacks i don't know if you guys know this they they do happen they're not as not as frequent but they do happen but typically it is a you know one domain gets popped they'll get enterprise admin and then they'll blast it out that same day to everyone this was way different there's the uh that event log up the top as you see that's the cobalt strike beacon likely an smb beacon if you guys aren't collecting tangent again that's system event log 7045 if you guys aren't collecting that building detections around that write it down put it in your system tomorrow start doing keyword searches on it if i if i was a detection person with no budget or if i was a responder and i can only do one thing it would be to collect that log it is that valuable in 2020 we put out a paper about detection opportunities for adversaries in 90 of all irs that we work globally we found evidence of the adversary in that log and that's built upon how things like interpreter metasploit cobalt strike any one of them covenant all those c2 frameworks how they do things like privileged escalation or lateral movement they all do things like by creating a new service right so fundamentally it makes sense because it works right it's the same reason why people use scheduled tasks all the time because it works collect that event log build detections off of it we're collecting our data going through two weird observations happen next first is that we're only finding evidence of the adversary within the server infrastructure nothing in the workstations nothing in the dmzs the only thing that we find activity was in windows core and internal infrastructure server infrastructure so that's weird because we can't identify what did they would they come in through hq why can't we find them did they go to west go to hq and back that's stupid that doesn't make any sense and the other thing we do is we get a yara hit for m.exe and it gets a yara hit for mimi cats we pull that back now on this is on a domain controller and on this domain controller was an av and it was a real av it's not like bob's av that you download off the internet it's like something that's going to many cats on disk every av is going to hit that right like that's baseline we pull that back and the only thing that hits is our yara signature we run it through our sandbox the only thing that hits us on our yard we run it through our all of our avs that we have nothing hits toss it over to the reverse engineers and we find out that it is a completely rewrote version of mimikats in fact the whole logon passwords module within mimikatz is completely obfuscated and rewritten and it extracts a go binary and that's how it ex gets the creds from windows holy right like that's way beyond the capabilities of most ransomware adversaries picking up the phone i'm calling people and i'm like we got something going on here this doesn't make sense we're going to have to do round-the-clock monitoring on this one the other thing we pull back is that sec url chk file start looking at that it's what's on the left here we start digging through that data and we find out that it is an arp it's a custom back door that takes a lot of code from power runner if you guys don't know power runner is it's a it's a tool that you can use to run powershell commands or scripts without running powershell.exe you pipe it through it does the dotnet calls directly translate them so you can just run regular powershell commandlets without invoking the binary so it's a bypass mechanism the other thing that we see is that it listens on our pcs we'll accept any commands execute them through a command terminal on the bottom there you see that function install what that does is that that's its persistence mechanism right so that is creating a new shared service so within windows there's two well there's four different types of services but there's two main ones there's standalone and they're shared process services now it's not unheard of to see adversaries use shared process services as a persistence technique it's very unheard of to see it from a ransomware operator right because you have to custom craft a dll you got to make sure it executes but it's in a really good sneaky way because you have service hose binary loading your pro so it runs as a as a library that's loaded by service hosts and they also tuck it into net services as a service group which gives it unfettered network access so they have some dev capability which isn't something that we typically see amongst ransomware operators it's usually rinse and repeat there's not a lot of custom tooling we start digging around on that golden that domain controller top right there event log four four or four six six two if you see that without that it's not a computer account name in the account name write that one down that's an evidence that that machine has been dc synced right through manycats problem is the associated user account is not from the hq domain it's from the east domain so now we have a multi-domain hop going on to deploy ransomware like now that never happens right that that this is the first ever i've never seen it again looking around at more of the data we can see that the adversary had created a golden ticket and taken advantage of the extra sid attribute if you guys uh there's a fantastic article that harm joy wrote it's called the truspocalypse he can explain it much better than i so if you're interested do that all you need to know is within active directory there is a attribute called the extra sid and it was created is either to whenever you're migrating content to a new domain or you're upgrading from like the functional level of 2008 to 2008 r2 something either one of those i can't remember but what he discovered and then mimikatz implemented is that if you own the child domain and you have the krp tgt hash you can append the enterprise admin account to your account and the parent domain will respect it effectively meaning that if you own a child domain you can own the forest that's a big deal right because now i think that fundamentally changes what you think about trust boundaries in terms of active directory and what you should be doing from a prevention or detection point of view is that any child domain can take a parent's privileges and so on and so forth right in this case it was the root domain worst case scenario so again we need to expand scope call up the east domain and they're like well we don't have a problem we don't have any alerts our we don't have av like avs clean i don't know what you're talking about there's no new admin accounts created we have we have no idea what you mean but you know ceos on the phone at this point they're like do the thing deploy the tech collect the artifacts start analyzing the data we find 100 days ago 100 days before ransomware that same custom mimikatz variant is in the east domain so 100 days like that's a