← All talks

I Knew You Were Trouble

BSides Dallas/Fort Worth43:3955 viewsPublished 2025-01Watch on YouTube ↗
About this talk
BSidesDFW 2024 Track 1 Session 3 - 02 Nov 2024 I Knew You Were Trouble This presentation unfolds a narrative of how a single OSINT pivot, a SHA-1 hash of a TLS certificate, unraveled a network of Lockbit Cobalt Strike servers. Beginning with an unexpected discovery during research on Blackcat ransomware investigation, the talk will highlight how a single pivot led to the identification of 44 related IP addresses and the nexus of domains managed by a single entity—shedding light on operational patterns, missteps, and mishaps. @jbeley Jeff has nearly 30 years of cybersecurity experience working with Fortune 500 organizations. He has led some of the largest nation state investigations - to include cyber espionage, critical national infrastructure, and cyber criminal ransomware cases and is currently a Senior Manager and Global lead investigator with Accenture Security's customer facing incident response team, working with Accenture's largest clients, to investigate and remediate latent and persistent cybersecurity threats. Jeff oversees teams of investigators and threat hunters, leading nation state cyber espionage investigations, threat actor eradication, and destructive ransomware response and recovery efforts. *Jeff's current role entails making artisan Taylor Swift GIFs, chatting with various AI models and consulting law enforcement on breaches of national importance and his team's work has led to a number of convictions of threat actors at the behest of some Accenture's largest clients. Jeff's incident response themed cocktail recipes are legendary pain relievers.*
Show transcript [en]

all right hello everyone welcome to the third Talk of the day we are in track one if you haven't checked in in the registration please check in uh make sure we know everyone's here uh hopefully Isaac's ordered some pizza by now but who knows we'll see here we have Jeff Bailey um I said Bailey right F oh my goodness I'm sorry Jeff fet Jeff F is going to speak to us he has nearly 30 years of cyber security experience working in Fortune 500 organizations he has led some of the largest nation state investigations uh to include cyber Espionage critical National infrastructure and cyber criminal ransomware cases and is currently senior manager and Global lead investigator

with accent customer facing incident Response Team working with Center's largest clients to investigate and remediate latent and persistent cyber security actors Jeff overseas teams of investigators and threat Hunters leading nation state cyber estan investigations threat actor eradication and destructive ransomware response and Recovery efforts Jeff's current role entails making Artisan Taylor Swift gifts chatting with various AI models and Consulting law enforcement on breaches of national importance and his team's work has led to a number of convictions of thread actors at the behest of some of accenture's largest clients Jeff's instant response themed cocktail recipes are legendary pain relievers let's give Jeff a round of applause all right I won't read my in this slide uh very covered it very well

I've been doing itd spots before I could drink legally uh I've very strong opinions on versions of thre said and op and the opinions that I share here are my own and not my warning there will be a lot of tayor slips first uh thanks to my wife she put up with more ransomware and more Taylor Swift than anyone individual should my entire team for putting up with Taylor Swift they they paid the deal rent where and uh dedicate my to pass so this is a true story about how we found an entire loet affiliate Network for a single TLS we liked it so much that we actually put it on a challenge course so we call

this the yeah 23 D7 act but we know by a different name and we'll talk about that in a little bit so back in February of 2021 we were doing some R um research on a black cat BR so a different gr than lock and we found some overlap in the infrastructure they were using the same as TLS and we were able to track them through P so we're talk about six different cases four separate enes four Industries and three continents also my company it should be seven because pure was hit by lockit back in the day but we'll talk about that so all of the inion that we're going to talk about today start off with

a sof goish or fake update the thing that says hey your Chrome browser is out of f excuse me and so what will happen is you get that fake popup it says you will install this and you'll get be able to view the site well back in 2021 to 2023 we had a number of what we call fake update Fridays 4:30 on a Friday customer calls and they have fake update or Rockit or something else so let's talk about some quick mitigation here one loocking ads I can't yes I can't State how critical that is to being able to uh uh to be able to stop Ransom additionally being able to change those default associations will definitely

stop the initial uh payload and then lastly printing on smart screen for Edge or for uh Chrome will stop the initial spread here A Brief History of lock so back in 2019 in September you see ABC ranser come on the screen uh they are precursor to lock bit we see them Rebrand in uh in September of the same or the next year and then we see lockit 2.0 released in June of 2021 uh I can't believe how wrong I was about this uh Lo B is just another answerer group um I couldn't find the exact quote of be saying this by re so now they have my attention I really wasn't paying attention to them

but uh once they hit my company uh you know they became on my list and read un so the first few engagements in our story so we have this Black Cat event back in 2021 nothing really interesting about it not locked it then we have our first lock bit 2.0 case we learned a lot about them but we didn't know enough to do anything about it that was until our second lock bit case and uh The Lure of the lock bit case was these shoes

and our girl Taylor really liked the the shoes there so we met well I say we met we ran across an initial access uh broker we've identified as G Bron and G Bron ran a number of registration domains for the S poolish and for the Cobalt strike C2 that we would see in our uh excuse me in our intrusions and from there uh one of the things we learned was they were not using protection or so we thought and we'll talk a little bit about how we were able to unmas the who is of G BLS here just a moment you know we thought they were just being cheap not paying the $261 or 199 rues uh to do the reg Ru

stuff so you can see here we have a redacted Gmail address for our domain Expressway products projects.com and with a little bit of magic we have gands so apparently what happens is the reg Ru quiz server is not fully reacting all of the all of the uh data points that it's bringing back and so when you do it with who is or virus total use it this way when you do it with passive total or now Microsoft Defender for credit Tel uh you'll be able to still do this I tested this yesterday it still works so if you ever run into reg Ru you can find one of the other things we found interesting with this particular thread

haer is we found a pattern in their domains you can kind of see there theor orgs are mostly St or Nets the docs are mostly calt strike except for that one Mage C which we think is an outlier just due to the age and you know they register a lot of scammy stuff and so we started working on a pattern here and we our girl Taylor brings up a good point here so one of the things that we wanted to look at was where are their patterns in the who is registrations by our our guy Gans B and you can kind of see he registers two or three a day uh and you know they follow the same

patterns as you know we saw our slide here uh but we see this Gap over here on the right side and we were really really worried that he had G bar or something like that well it turns out the 24th of April is a national holiday in Russia so it's Parliament day so they took the week off and that you resume a little bit later in the month so talk about the domain analysis here so cryptocurrency and XYZ TLD almost always other SC d new things uh the high entropy and net.org those would be the sou goish and here right in the middle is what we call The Sweet Spot Cobalt stri most of the time

those IP words J Chrome Etc and then the words like uh mashed up Expressway projects.com they put two or three words together and they always dump it into the KD so one of the things this particular analysis took a long time to do so one of the things I learned as a result of this investigation was how to use syx and essentially this saved me about 30 CPS every day that I was tracking This Thread out and basically it would go in look for our Gan fron friend they would look up the who is we would grab the passive DNS and if it was hosted or parked at reg Ru we said weal prayer and

so so whenever they move it to selectel or somewhere else we would then start seeing we use it in our client environments so we did you know start understanding a bit more about this particular uh lock bid affiliate um to include their phone number yeah uh we tracked this to a residential apartment complex in New Jersey um and this might have led to a few fun late night phone calls uh this was not an excuse to call them but might have done that a couple times one of the other things that this information uh we took this information uh to visit the the FBI and then just a few weeks later uh we have our one of our first uh

so this is an updated version of the of the posting you can see the date there so it's a little bit updated so yeah we went in and uh you had some fun there so let's talk about everyone uses Cobalt stke what's different about how lock does this so one some of the unique things that we found one uh as we all know Cobalt strike can inject into any you know Windows process you using the sacrificial process well lockit is the only thread actor we have found that uses wfal to eject so when you see injection in the WFA it was almost always locked in uh one of the other things we did was we were able to get

the Cobalt strike Watermark Fe and this that basically the license key for this and was able to help uh send that to our good friends that so does that mean it was liced or was it just we think it was license we think that there was a shell company involved here and then they you'll bought it from for uh they ended up using the blister loader which is was a kind of a pain in the butt for us to be able to unwrap but uh my team of reverse Engineers were able to pull that out and then they use masquerad file names why what I mean by that is like win image. XE but it's in

the Pro rooted program data versus you know where it lives on the Windows system and then uh one of the other unique things we found was their use of a vulnerable zcpu driver so they would load this driver into RAM and exploit it gain privileges to be able to disable antivirus be able to you know grab the keys without you know really trying there so there were a couple of good offset fails one uh when they set up their new c2s uh they use the default certificate twice not cut up in public reporing we show you that in a minute uh the lack of who is privacy we showed you that and that same U and oh the burmer

phone number that was used I forgot to tell you this was used by media land bulletproof posting so this is all together with them that Y at and it's all one you know thing together so once we had all of the CPS I decided to put them on a graph to see if there was anything interesting and what I found is a couple things so first uh you can see those arrows there those are the times they screwed up and used the default CT what we call the 6 ecert or Cobalt strike and they got caught in public reporting and took it down within a day but what you can see here is that the pattern is every five to six months

they totally recycle their their INF leaving one there that they could migrate their victims from from server to server very easily using the native features of 12al strike I love overlapping intruding sets it's just the greatest feeling to be able to do that so obsc WIS so they were able to use your know tooling to obscure their infrastructure my knowledge very few people have this particular uh understanding of this particular affiliate uh they were using some tools that kind of annoyed me so one of the things that I like to do is I like to inmap them and grab the beacon which makes my life easier for doing reverse engineering and things like that well they were using something couldn't

figure out what it was exactly maybe something like C2 concealer that prohibited all of my inaps from all the places that I tried from doing that additionally they were using Tactical for you to stop me from hitting them from uh VPN providers and things like that so they did a lot of things to do that by the way there's no tailor certificate for this because wi pants so we decid so we mapped all the c2s and you know we use the altigo there we store all our data in a tip and you can see over at the bottom left hand side is the certificate that they used and everything is kind of centered around that uh we I got to the point

where I memorized some of the Miss by ID numbers and so I was able to get quick to say oh let going 1256 and you know be able to you know move these things around um one fun thing uh when we share these things with the FBI none of them are on teams or anything kind of uh you know computer they're all on the phone there's some kind of Bureau regulation where they're not getting on the Keens to share tell them locally and then share these things later with and dazzled into the power I can tell you this infrastructure will go down in CL so back to our story so in June of 22 uh we did a new decryptor

so we have lock read allo um and then one of my clients has the worst thing ever ever they got hit by lock twice in the same day in two different parts of their company yeah it was a really bad day for uh they're one of my favorite customers and they love us now but that was like their worst worst day oh by the way lb3 and four is the same company lb so they got tce in the course of the year so they're better now they're better so we're going to talk a little bit about lock case five because this is we first time that we know enough about lock to actually stop the encryption and

stop the data ption however our client declined to let us to go to the FBI this time privacy reasons you know it's embarrassing you know it's all comany culture but I promise I will shake it off so let's talk about this other event timeline this is the time five our good friend Jimmy Garo there was The Lure and you can kind of see the standard Cobalt I'm sorry sou goish the Cobalt strike move there I'm not going to drain this but you know thread actor was able to move laterally within 10 hours off the Box you know start withing up files D the get D elass and then on the 31st we have to Cent put in a block and

everything stop everything yeah I was I was really happy that day so one of the other things that we like to do to kind of understand the enormity of these cases is we make a loop craft and you basically show all the places where the gra after has been and we can kind of walk the through the visual representation so that our clients can understand oh my gosh you know 427 different servers or you know keep something the else at here and you know here's the end of the the intrusion so it's a good way to show them the timing of of these kind of things I would shake it off so the day after the FBI uh you know

up I went to go see Taylor Swift it was amazing had a great time so the next day so because our client deined to let us go to the FBI back in March of that year we needed to recreate the resarch based off of other facts and things like that so fortunately we get this is the last L bit case we we ever did well where there was encryption so uh fortunately we able to use that data the client was more mediable and towards the end of the month we were able to to uh stop our research on that it was a lot of fun that research so one of the things we do is we track the the number

of victims of of various thread act groups lock bit you know had kind of a good run there and towards the end of May um we'll talk a little bit about what happened there uh they started reposting victims in an attempt to uh regain notoriety and public you know view of them uh but they had another problem so back in March of 43 uh we go back to the FBI and uh we have some good results there uh gu arrested in charged uh you know with those attacks um and then um what's a little funny to me was a few months l or actually eight months later or Not Sorry 10 months uh 10 months later we see a new cobalt strike

CT with our same PLS C uh after the guy got arrested where someone has the kit out there we still see this pop up from time to time nowhere like before fortunately um we were able to work with our friends at I won't name the place uh to to squash this this one very quickly so within you know a day or two of us contacting them uh we we saw all activity to that CQ stot so we all know what happened on the 18th of uh excuse me 18th of July this year and you all know what happened on 19th the 18th of July uh you know those two guys quite guilty and yeah and by

the way the FBI was absolutely not one so why am I telling you this story uh it is a lot of fun to dunk on thread actors however you know one person and one team can really make a difference you know you can too you know it didn't take fancy tools or accesses we what I showed you with public versions of those tools so you don't need a lot of money you don't need a lot of access to affect the change for these things uh information wants to be free you know we we do better when we share both our successes and failes as a as a community we have to do that and then law enforcement is here to

help and uh as we saw they don't often questions from your experience is this mostly all yes malicious ads yeah most of these came from like the top headit of Google you basically someone said you PA Jordan cmf 111s and they click the first L Jimmy Garo C the first link that this is why blocking ads is so critical

no by yeah well by policy I can't the you had analis so I know you right

this is y from a company called y works and essentially you um you can take an Excel spreadsheet and load it in and basically it does this automatically yeah y y Echo also this doc this one here also NE so it's very customizable and it's it's great so what yeah once again up

there yeah like lock bit case three and four uh we were done in about four weeks so and that was kind of a large one you know two entities uh we actually split normally do the global team rotation we actually had to split up to make sure that the team um in Asia in Asia had the right number people so we had to kind of split up on

that so my my viewpoint is a little bit skewed here so as I mentioned I do work C clients which is normally large corporations so I don't normally see smaller organizations however in tracking lock through all these years we have noticed that after the big fall that it's only smaller organizations getting hit I was going to ask you know like going back to your link analysis you know so many systems we pulled out and everything else are you guys using special tool like I know right now final to cover the network like that or we have our own up and spoke tool it's very simple similar to B analyze that we created' been made the last few

years you does same grabs data brings it into some kind of presentation L and then puts it analys on the webs makes it really easy

detrimental something well so there's several schools of found on that one one uh you've seen you know the red SE blogs everyone talks about how to do this with Cobalt strength so the barrier to entry is is lower there there's all kinds of documentation around that um it's also because of the ubiquity of of cobalt strike it's sometimes it's hard to tell whether it's a red team or a bad gu and so that's one of the other reasons they kind of blend in with the noise is low those are kind of the two reasons I think they use it um I mean ap28 used that during the solar winds breach it works it's extensible it's

it's a great

tool now it's called

defend say that again yes

yes uh I will say tdpr kind of killed who is hunting and I I love I love do this privacy but yeah so essentially what happens is the way that c toal is doing this they're basically getting all the raw data from the source and but if the filters when you look at it with regular total other services other

than

stim yeah they were using an active EVR solution but the thread actors disabled it so they used that zcpu driver I talked about that um disable ad and EDR on the Ino so they were they were blind and unable to stop uh between all six of those cases you probably will see them all I I won't say the name I won't say the name

yep y they they were doing b y v d before it was cool any

questions yes all of those companies are my partners so I can't really speak get a review of this tlor no I I have not and I I might need to say that I am not sft on SEC say that a few times cool any other questions any other questions about incident response or things like that I was going to so Tech Emily gave a good completely different typ of um when it comes to doing IR you know yeah there's the Big Technical part of it but what do you think for some of the best resources for the the crisis management aspect of it keeping the uh the organizations you're working for you're coming in as consult like hey

we've done this before you're going to be okay so for for me what's been really helpful is be that calm voice in the you no matter what happens client screams at me this has happened more than once the client is actually screaming at me and okay you're having a bad day let's step back from this and to figure out how to get you back to making Widgets or whatever you need to do so the other thing is organization is param up param up here you know not just the ioc's and TTS and what we see on the live but you know when do we start recovering what backup do we start recovering from and basically having

that full story of this because guess what The Regulators are going to want to know the CEO going to want to know and there's going to have there's a lot of communications that's going to happen as a result of this so there has to be a good Communications plan that talks about just the facts those suppositions and you know wanting to make sure that we're telling the story at actually another The Story of Us what what is your Communications CAD we defer to the client however we suggest twice a bit because any more than that you know I I your investigation I I say I can either I can either status you or I can leave the

investigation it can't be both at the same time so you know uh and it'll be like twice a shi so depending on the client will be Global operations and you know so on your team are

no documentation is everyone's responsibility so yeah so now we do have people to do like quality insurance and engagement management who have more of those soft skills uh than our their investigators because you know I like to say I like to keep my investigators home and in their sweat pants that way you they can work at their own pace and have their you basically be able to live their life in some form or fashion while still doing an

activ yeah so fny you should bring this up so for for cases two and the grouping there um we kept all the recommendations and we basically recycled powerful and so these were the ones from the last time and it was really impactful for them to see okay we sat on this for six months or eight months and we got hit again and so there was a big culture change within that organization that started happening there funny thing is two or actually several places Happ first um on case number two the IR Le quit the day it happened he he says I'm done and the SE so the deputy CA and you know his team were leing on us very heavily so right

now the guy we promoted from depy c c as a result of culture C that we helped with over that year um funny also not funny so much of what we do is kind of shot uh uh they were flat for the yearing they have no profit because of all the yes and they still promoted the yes back

the how do you protect your own team so I have another team that protects my team and basically you know they they watch our Dev they make sure that you know they have all the Pates make sure they're not talking up to B guy.com things like that so you in addition yeah so my team is just investigators we have people who support us do recommendations mediation all those things yeah we're basically from point of impact until the investigation done and then we work the transition other teams so you recy those

no no it goes back to kind of P empathy to to yeah I mean you know we we we poke one a little bit in Charlie but yeah it's like oh my God company what again yeah so it sounds like you do a sounds like a bunch of big companies but probably haven't had it in a while but when's the last time you saw shut off your network oh last week okay so how how how's your team responding remotely when the network having we yeah well so we do a couple things we have the if they want us to go on site we'll do that although that can be expensive but what we have done is

basically give them a way to capture images and capture you know triage data from the system and then sneak so we have our own little collection tool fits on the USB drive runs in a few minutes and then we just basically take that s files and then we

process so yeah great question so my team is say about 150 people GL about 40 50% of that is actual investigators Etc uh the remainder we got you know managers we have people we call engagement managers basically like a think like a project manager but with so much more authority to to take and make things right and then we have you know people who build our tools people are sh sharp and shiny so yeah so basically a full team and then outside of my team because I work at a large corporation we have another group who does launching another group does Readiness Services kind of before the impact so we cover all the things my

team Focus only on IR so okay I know you mentioned like engaging with the FBI what you have like oh one of our shut down now FBI for that yeah great question so it's mostly based on two things one is can we provide actual intelligence to the FBI to be able to say it's this guy or that guy uh the other kind is you know the F allows us to basically give them all of our information that we have and they can use that in their yeah it's really dependent on the client's appetite but you know are out I have a I have a contact but yeah my my my current boss is the former

assistant you can call ic3 and then you say you know I want to speak to someone office you yeah uh New York office is having a lot of cyber crime rning now wor thing you could is called

all of those in one so so sometimes you know the client will have like break glasstic out for us and then we'll activate them quickly so they you'll log into their EDR or to their scam or do something like that uh some clients don't like that and so what they'll do is we'll either uh run our tool or run you know a collection tool and then send us the data and then we process it and then it's been rare to go on site before on site about 40% of my life uh since Co 5% and it's only we've only been going on site to do like

post so if I was to break down my cases over the last year 80% ransomware 5% uh are actually maybe 10% business email compromise 5% nation state 5% Insider threat I will say Insider threat cases are some of the more interesting ones um where you have to really deconflict normal activity from that the I so we had a case where pharmaceutical sales rep had his phone taken through a domestic situation so his partner took his phone knew his password and started you checking out drug samples that weren't you know legit so federal crime uh she oh this person uh sent really really vulgar emails to uh to the boss it's it's a really weird pornography so what

we had to do was deconflict the activity so he was in this town on ISX she was in the same town how do we de confed it's the same it's the same ASM and so what we ended up doing was we M the entire ISP of that c of that City and say when you're coming off of this rer leg it's bad person and then you know our God and so we were able to do that to help deconflict that activity so because it says it's you know our our victim here but it's not

so with very few exceptions I haven't seen any public reporing on any you know outside of SEC yes yeah

yeah we we have seen it up taking that so the Air Jordans on wearing they were my my basically my victory dance on the thread actors and so every time since then I've gone to either speak to National international law enforcement and I buy myself a new parent so in in the last two years I bought nine more pars

if 80% of casere how many of those companies are will to out so my policy I can't know I can talk about it but my team you can't even tell them to you tell them quot talk to The Negotiator you step out so you and far um now one of the things we do once we do that is we get the cription key we make sure it's not going to be worse and make sure it's going to be I think I've only dealt with key like five or six times and I've done 170 of these in the last eight years I was say would you agree that all the effort goes into

performing like if they would put half as much effort into their as they do everything goes into it'd be great no we tried to reverse the inors and get the key out but it's and they're not it's not meant for Speed I've got one announcement for you all but first let's put our hands together for jeffa [Applause] [Laughter] [Applause]

announcement um so now it's lunchtime up until 13:30 1:30 that's when uh the next talks and uh workshops begin there is a University cafeteria if you go on Google Maps and you're walking at the University cafeteria they're going to get confused because orientation going on right now and all the orientation students they have like a free meal ticket they're going to ask you for your ticket you don't have a ticket so if you go over there just tell them you're paying cash tell them your amount part of orientation there's a t of other restaurants nearby

than thank you than you