Tim Crothers - Analyzing Public Breaches

all right well thank you mark that is the most unique intro i've had yet that was that was fun i feel like i'm on johnny carson or something all right analyzing public breaches so uh if you came today hoping to hear some inside dirt those of you who may know where i work and have had the pleasure of working i do occasionally get some inside scoop on stuff that's not what this is about sorry to disappoint you will get one probably factoid straightened out here today hopefully but uh but that's probably about it for uh inside information on public breaches this is actually the opposite uh this talk came about last november right solar winds dropped

mandiant you know was releasing all of the the stuff on it and and i see peers and teams talking about a little bit but like a lot of it was just blowing right past literally one of the most significant breaches in recent history in terms of implications for us as an industry things we can learn from it blind spots we can find all sorts of tidbits in there and and a lot of folks were missing it and that's when i realized that um you know i've made a maybe a hobby but maybe arguably a career out of studying public breaches because think about what we do as an industry right our job is to defend our

organizations i've really come to believe that as a defender cyber security is just a constant exercise in prioritization what do i mean by that we can't block all the things so our job is to figure out what are the things that we need to be blocking right now and nothing against all of the amazing stuff out there about you know we we see we come to great conferences like this by the way fantastic this year as usual can everybody give it up for the volunteers and organizers of besides augusta this is one of the most densely packed in terms of valuable information all year and frankly is is one of my favorites to attend but but so we're getting great stuff at

this but a lot of this stuff is coming out of research which we need and we need to figure out all of this i just saw a talk last week for instance on how do we poison ml based malware detection right really interesting but mostly theoretically not something most of the cyber criminals whereas if it was a breach last week that meant a real cyber criminal did something that means it's real data that we can use to go oh where would we stand up in this now as you might imagine there is of course some art to this so i'm not going to be able to exhaustively cover this in an hour but what i hope to do is really

stimulate you with some key ideas and some maybe different ways of thinking about this problem that'll help you start breaking down because it's a free source of intel that can literally help you just have a really really strong cyber security program wherever you're at so we're going to talk a few things one why why in the world is this worth doing how do you go about it maybe some lessons learned the how i'll talk a lot about kind of key questions because i think that's the heart of it uh and then some common mistakes and of course uh hope you know we'll leave some time for q a so let's talk about the why and and

i i've even got a prize for this one so what brave soul some of you know i just spent six and a half years at target i had this little breach back in 2013 pretty widely publicized i doubt there's many of you in here that don't know about it right what was the foothold for the target breach in the back hvac system who in here agrees with that who who says yep that's that's what i've heard nope it wasn't you still get the price uh i set you up there a little bit i used that because here's how that happened i'll be back right back recording there you go sir absolutely so um so what happened with that

was news breaks target breach target's got this huge brand you know target etc right so so everybody wants to know what happened and in the comments on brian krebs blog uh there was a a post you know about um well there's an hvac vendor that was breached and just disclosed their breach who's also a vendor for target and over the course of a couple days that went from a comment about you know coincidental timing to becoming the source of truth for that was the foothold for target's breach so that's the one inside information that you'll learn today that that was not indeed the foothold but i illustrate that because that's a great example of the why why

this is worth doing is first and foremost most of the information that is publicly disclosed is wrong and it's wrong for a bunch of reasons not all you know malicious or anything like that it's just companies are not incentivized to be transparent about this again let's let's take the target situation the their entire board of directors was under lawsuit all of the state adjourn attorney generals uh the fcc launched three different lawsuits two of which came out of misinformation that was dumped potentially by some insiders etc right and and so the unfortunate reality is much as many organizations would love to be completely transparent and completely uh forthright about what happened they can't be you know if your public company

uh and you put this information out then what happens is you know the the lawyers that are suing go oh well why weren't you doing x y or z right and of course we all know we can't do everything so not excusing anything that's not not the intent here but this is a a real big piece of why this is is uh difficult to do um the other thing is that you want to remember that these organizations are trying to balance speed and accuracy in the reporting right there's a cycle that goes on with this right where the breach happens oh crap we call in some sort of organization often to help us do the investigation

bits and pieces start coming out in bits and drabs because we never have complete and and it all just ends up getting confused in the in the media so let's pivot though so why is it worth doing well i already gave you a hint on some of that like i said i very much believe that our job is to have the right controls in the right places at the right time and so if that's indeed the case for us right um knowing specifically what the real world actors do right versus what you know nist whatever says we should do and again that's not not a knock on nist how many of you let me comment this from

another direction how many of you have been in a in a meeting where somebody said well could they uh cyber criminals potentially leverage your security platform to ex-fill critical data none of you've heard that i see a few hands right or some variation on that of course they could right our security tools are used against us regularly by cyber criminals again if you if you follow a lot of the breaches but if you're a cyber criminal why go to all of that trouble when you know i'm sure all of you seen recent pieces where a lot of the ransomware actors are offering cuts on the ransom in order for employee insiders to deploy the ransomware internally for them

fishing is still the predominant method right if you're a criminal uh the way i the analogy i like to use for cyber criminals it's not that they're tend to be that more sophisticated or smarter than us but they're more motivated than us in most cases right we get in the job and yeah we're passionate about our job and etcetera but we collect a paycheck every you know week or two weeks or whatever the rhythm is whereas the cyber criminals are kind of like 100 percent commissioned sales people if they don't successfully steal they don't get money right and that adds this level of motivation where they keep trying keep trying but they don't care about the

sophistication or rarely i should say right they just care about success and so having those insights into what's actually being used is incredibly valuable often there's really significant gems in there that we can find and i'll i'll give some examples of that and my last reason that i would suggest why this is worth doing is because the threat actors are doing it if you don't think the cyber criminals are looking at situations like solarwinds and going hey how did those folks stay under the radar so long what techniques did they use what can we introduce into our toolkit right look at all the ransomware actors that are just copying each other in terms of their

their tactics so even if you don't think you're necessarily going to learn something valuable i would argue it's still worth doing simply because of the fact that there's so much in there this is wall street journal august 27th of this year this is not a good look for us folks right the the hacker that claims uh john bins that claims um to have be behind the t-mobile breach did an exclusive interview he later did a couple interviews with a couple other orgs as well with wall street journal and and that's their key quote their security is awful now i i don't know anything about t-mobile's internal security but i rather doubt it was actually awful

right they're a big company there's no way they aren't investing tons and tons of money and and we don't want to be this right not just as an industry but of course as our organizations this is my kind of final graphic why this matters to to make the investment in time into analyzing these so let's move into the how because the this how is much more important than the why the how uh i said earlier i think it's really important to to start with realizing that there's a cycle almost a variation of is everybody familiar with uh gartner's hype cycle curve right where there's a new technology and everybody's all excited about it and then there's

the trough of disillusionment where everybody goes this doesn't do crap and it's all just sales and then everybody figures out how to use the tech and it's somewhere in the middle right that's almost how breach investigations play out right oh no the the sky has fallen they used sixteen 0 days and now right and then oh no this is no big deal and then somewhere in there when it all starts to finally come together we realize oh no there's there's some interesting pieces in there so while you're doing the analysis one of the things you want to bear in mind is how close are you to the origin point of the breach notification because the

closer we are to that the less likely we are to have useful data it's only in the collating that over time and starting to analyze it that we start really getting some some useful conclusions out of this and so that's a piece and then i think this is other reason why we often see so many uh gaps showing up right because we as human beings just wanna naturally fill in gaps uh i was at a schmuck on talk oh maybe 2016 and i literally had to get up and get out beca uh walk out because the speaker who i'm not intending to criticize at all you know had somehow come to the conclusion that uh that um

you know the malware was deployed at the target breach using microsoft's uh you know management server right which obviously was not the case or i know was obviously not the case but it's frustrating when as an industry we propagate misinformation and and don't go back and check and verify so that's really what i i want to delve into so the the the heart here is let's talk about key questions that we should be asking first how did they get in how did they get a foothold well let's take some of the most recent breaches as examples so we'll use solarwinds t-mobile and colonial pipeline now i want to reiterate i am not going to disclose any this all of this is

based upon publicly available information um so solar winds how did they gain a foothold well they you know uh breached the company solarwinds company managed overtime uh hopefully all of you seen the timeline for this so we're talking late 2019 all right and they didn't start deploying the malware in their production until march-ish 2020 uh going off the top of my head here rather than the the full timeline but this has all been publicly shared and is is readily available but more interesting to me because i'm mostly concerned about the end companies that were being targeted by the actor behind solar winds right the foothold there if you looked carefully they immediately pivoted off of the

malicious binary inside the solar winds distribution to an entirely separate piece of malware so we had two pieces of malware being used that was immediately interesting to me right and if my initial thought was hmm i wonder if that's so they can evade detection not enough data to conclude that based upon that little bit of tidbit but i share that because that's how i start to collate this uh as i'm looking at these i go okay how did they gain a foothold and as i'm internalizing those details i'm asking lots of sub questions of well what does that mean what does that imply what could be the reason behind that oops before i go on so so

uh colonial pipeline with colonial pipeline it was disclosed that they had a single factor vpn right in play that was used with some compromised credentials um t-mobile t-mobile is even more interesting because we've got both the organization reporting and we've got an actor who claims to be responsible john benz who's giving his version of what happened so he claimed that he gained access in the wall street journal article through an unprotected router which is pretty ambiguous right uh in terms of of details when the ceo reported later he specifically said it was through a test environment that was exposed on the internet now this is him reporting after an investigative firm has helped them with what's going on

later in the wall street journal article with john bins he did say something about production test environments so i tend to think maybe the the you know the vagueness of the uh you know unprotected router maybe there was a dev environment that was exposed uh to the internet and from there one of the other specific that comes up is in is in the next question how did they move around inside the organization so before i pivot back off at t-mobile specifically john bin's claim that he found credentials well if this was a dev environment right i i'm sure none of you have environments uh where your your engineering teams have hard-coded credentials in the uh in the software right not not

a thing that happens so dev environment might see see again how i'm i'm trying to tease apart the implications for what might be the root uh issues of this and so john bin says he was able to find credentials after his initial foothold that eventually he was able to leverage to access an oracle database server that had the customer in again i don't know how factual any of this is i want to be clear here what i'm trying to to tease through is data that we've been given and what might be the implications for that okay uh the ceo didn't say anything the t-mobile ceo didn't say anything about the lateral movement phase beyond just

access to a dev and from there was able to pivot into production environments not not in terms of a lot of details there what about the solar winds entirely well not maybe entirely new this has been theorized for a while but certainly new in terms of widely used the golden saml ticket if you've not gone out and and dug into how the golden saml attack works then that's your homework for today first thing you should be doing especially because in the the golden saml attack that was used as part of the overall solar winds with many of the organizations not all and that's an important distinction because the how did they move around inside the

organization solarwinds becomes really really interesting because when you start looking at the actual end organizations that are targeted you find variations the same things didn't work or weren't used at least in every one of the breach customers which then makes me wonder why if the threat actor was using gold and saml at most of the instances based upon the public data why weren't they using it on all of them see how the teasing this apart is the the heart or maybe the art of of doing this okay and then what about colonial pipeline colonial pipeline was undisclosed as to how the lateral but there the interesting bit is is it was disclosed that the initial foothold

was april 29th and the ransomware demand was may 7th so eight nine days later pretty fast so whatever they were doing lateral maybe if it was that single factor vpn maybe they were able to leverage those same credentials i don't know right again i we want to be careful here we want to take the data that's given and that was disclosed to try and use that how is this going to help and there is a how does this help us coming i promise another big one for me personally is what tools and malware did they use malware tends to be very personal to cyber criminals now i realize this maybe isn't as useful for everybody here because

best value comes out of the tools in the malware if you've got some reverse engineering skills but if you don't that's fine too most of these for instance if you look at the solarwinds and i'm give a uh uh article uh or url sorry at the end with lots and lots of data for you to go look at so you can practice some of this that you know takes apart the tools and really breaks down what they did how they work all of these sorts of things and in the case of solarwinds the tools and the malware i i don't think there's a better word than elegant they were just elegant the level of craft was so

obvious in looking at you know very small piece of code with very strong evasion and capabilities um and and so similarly you know what are the tools in the the malware then i like to ask how long were they in the organization before they were detected why do i care about how long it's a pretty good indication of the adversary right this at least the skill level of the adversary involved if they were a long time in there undetected like the solarwinds adversary that takes a high at least unless they're you know as long as the victims aren't just mom and pop type you know really small businesses if they're going for enterprises of any size and

they were in there for a long time undetected that's a pretty decent skill level required to accomplish that right um conversely how fast did they move right did they get in and then very quickly that can also be an indication of high skills or it could simply be an indication that they've got well-honed well-polished capabilities right so in the case again months to years you know some of the victims for solar winds for over a year uh colonial pipeline was about uh nine days and um interestingly colonial didn't detect them right the the ransomware notification is what set that off uh similarly t-mobile effectively did not uh detect the adversary until reports went up about

customer data for sale on the on the dark web but time frame wise for t-mobile was about a month from the early indications to to when it all became disclosed and then similarly how were they detected right uh i already i i think the most interesting of these three is solar winds and if you caught uh kevin mandia's testimony for congress solarwinds was not caught with a technical thing solarwinds was caught in mandiant's environment because of a process a manual process whereby if any employees added a second mobile device to their duo dual factor a multi-factor they manually called the employee to verify that they had added a second mobile device so the the threat actors for solarwinds

were effectively caught because you know they successfully added a mobile device to the duo multi-factor for an employee the sock called the employee and the employee said no uh she was actually out on maternity leave at the time uh and so had not added device that was the initial clue that led to all of the the what we eventually disclosed again talk about a gem probably pretty sophisticated actor based upon the elegance of the tooling months in their most of their victim environments undetected and what finally outed them was a process hmm i might should probably go double check the the thoroughness of some of my manual processes in my environment but i'm getting ahead of myself

what was their objective often this isn't obviously told what's been disclosed so far for the solar winds in public data is that they are data of interest to nation states that's that's what we've been told publicly so far with the solar winds with t-mobile obviously selling the data but john bins in the wall street journal article also mentioned that he was looking for notoriety he was looking for a little bit of fame as part of that and oh by the way i also wanted to sell the customer data and get some money uh as a result of that um in the case of course a colonial pipeline ransomware right it they wanted to ransom uh and and to get funds out of

it right now again why do i ask things like well what was their objective because that's a useful tool for well is this something that i even need to care about then in my organization obviously we're all potentially subject to ransomware but we're not all potentially necessarily subject to nation state adversaries which certainly strong indications are regardless of your personal views on on the solar winds as a possibility there and again this is this is a useful thing for us to consider so i start iterating through all of these questions really with the intent of getting at well what lessons can i learn from this situation how can i take and learn from i i'm still from the old school that

organizations that get breached are victims not that i'm denying we have a responsibility to protect our data but i think sometimes it still gets lost that these are victims right one were there any systemic failures at the affected organizations what in the world do i mean by systemic failures well i'll often ask questions like what preventative technology was being used was it working right in light of what i know now if i understand the malware and i understand the the tools one of the things that you can do with virustotal is run queries based upon dates so if you've got the hash of the malware you can go to virustotal and go hey what technology detected those tools as of

the date it was in xyz organization and you can get a pretty good idea of whether the preventative tech should have you know uh helped with that or not um or what didn't work right and if if obviously there was a failure if they had a breach their preventative technology fail well why did it fail how do we get at why didn't they work and and often this is again where having a big network of of friends in the industry i i feel like our industry is uh you know i'm sure you're all familiar with the six degrees of kevin bacon right i feel like our industry we only need like three degrees to to know somebody who knows somebody who

worked there or what have you uh is feels like is often the case right and and figuring out this why did it fail now again we might have to theorize here my q key ask is if you theorize nothing wrong with theorizing just make sure you label it that way that hey this is a guess this is a theory just don't propagate it to friends and peers as facts right in in terms of of the why did it fail right often um when i'm able to to dig in and to get at some of these uh route why did it fail more often than not organizations have in place the right technology but they had broken processes or the

technology had fallen over and wasn't reporting to the sim like it was supposed to be or not and you only get at those by digging through these sorts of scenarios to try and and surface these lessons uh ultimately being i don't want to get burned if i can learn from somebody else's then i i teased at this one earlier with the golden samuel did any organizations defend better and if so why well i was in a conversation with microsoft early this year is actually microsoft's head of identity access management and of course they were front and center for a lot of helping customers respond to the solar winds and i asked him this specific question say hey did

did any of the customers defend better and and if so why and he stated that the reason why uh the yes that several organizations the threat actor wasn't able to leverage the golden saml attack because they were storing their root ca for the saml environment in hsms and when the threat actor was not able to access those because of them being protected in an hsm hardware security module that shut them down and they had to go alternative routes aha i better go make sure all of our root certificates are stored in our hsms and protected appropriately see where these gems start to pop if if we ask the right questions uh from from these sorts of situations

um and then ultimately of course what should what i'm getting at here is what should my org do uh in order to uh you know do in light of what is known or you know another way to ask this is what would we have done uh if we were in this situation right what should we have done differently and just start piecing through those pieces you will be surprised at some of the really interesting conclusions that you can find right and so much of this is simple okay single factor vpn hey when was the last time we validated that absolutely every one of our internet-facing vpns has multi-factor enable and is only accessible via multi-factor

when was the last time we validated we have no dev environments connected directly to the internet right we we take and flip all of these tidbits and we can leverage them for we know threat actors are doing it right now being successful right now again other threat actors are going to watch this success and leverage it taking that and prioritizing those sorts of things uh over maybe some of our more theoretical type you will often find broken processes broken things um you know think about the the uh the solar winds hey if a threat actor was able to enroll a device in one of our team members uh accounts do we have something that would

detect and validate that right that we could use to to realize that that that's what's going on now of course there's a ton of common mistakes uh i i've alluded to this one already propagating misinformation this one's tricky i'm you know i i'm not trying to uh well maybe even better way to say it full transparency is there's no way i've not also been guilty of this at points in my career right it's maybe an inevitable trap but the more intentional we are about trying to revalidate our information trying to make sure that we're not guilty of of that propagation of misinformation reduces the likelihood right start with a lot of that early public information

and you know that's often the least varnished and then compare that against the later information and the context is incredibly important right think about when i was talking about earlier how the t-mobile ceo right knowing the context of he said well xyz company finished our investigation and this is what we have learned right so so we know that he's got a lot more data we know that it's further on so probably higher confidence that we can have in that data and then how does that compare against the earlier data is a good way of of kind of trying to dissect the other thing is often we get hung up on on important tidbits you know the the the piece i uh you know

kind of uh set the poor gentleman up for on the on the the target breach with the hvac entry point right does that really matter to to most of us you know maybe we do have managed hvac maybe that's something to look at but a lot of those details that i think we as engineering type personalities often get hung up on often aren't aren't necessarily relevant and and the better i have gotten it not sweating some of those particulars the the better i found is the source information internal or directly connected to the affected org of course right the context of who's sharing it is is pretty relevant to the likely access now the key here being that if it's

anonymous insiders be really careful with that often people have a bone to pick with their organization and will take advantage of moments of crisis to spread misinformation or um some of the misinformation i have seen where i know is misinformation because of inside information maybe there was some notoriety that was being sought and some embellishment occurred right so if the insider uh is directly connected or they're somebody that is sharing their identity sharing the context and why they're sharing it that often is very very credible source of information for us uh we just need to be leery of say tips from underground cyber criminals um there's agendas right people have agendas and so we need to factor that in

uh and like i said already just a simple question of does it really even matter on a particular piece i think the other big mistake uh i see most common is really simplification errors so much of this wants to be a quick sound bite and so you've got media who are very talented intelligent people but aren't cyber crime experts right they're trying to wrap their head around it they're trying to portray it in a way that their audience can consume so they have to do some simplification well if that simplified version becomes the basis of truth then then it it becomes problematic for us and of course flat out assumptions right we've got to be careful about

assumptions again you're going to have to make some guesses the key here is label them as guesses as you're doing the work what i tend to use for doing my analysis is simply mind mapping software i'm just a huge fan of like mind node or some of these tools where i throw these pieces and parts right so i'll often start with those key questions that i i outline and i'll start throwing the data points up under them and kind of refine them and then start to interconnect them and it's in those thinking about those implications and those interconnections that i find really really interesting aha moments those kind of light bulb moments frequently so frequently that that

you know i i'm doing a talk trying to encourage all of you to do this and with that i actually have a challenge for you uh so i'm gonna give my linkedin uh profile here at the end when we move to q a but wikipedia has done an amazing job of trying to collate the solar winds there are the solar winds is a little unusual in that there is so much information available that it's daunting uh it can be at least daunting what i would encourage is for you to try this out yourself because like every other skill in our field the skill really comes from the doing if you go to this wikipedia you will be

slammed with a ton of information some of which is wrong wikipedia is an open source they do a great job but they're crowdsourcing the data right just like everybody what i would offer up as a challenge is spend some time reach out to me over linkedin and just say hey here are some of my conclusions and i will absolutely respond to you with right right wrong wrong right or whatever it's a good opportunity to practice where i may have some inside information i do which i cannot disclose for for lots of reasons but i can certainly uh uh you know help validate whether your conclusions are right or wrong that's how we as an industry in my opinion get

better and better at this is you know coming together um and and helping each other figure this out because this is how i think we get really good risk based security right when we've got that deep understanding of our threat actors uh et cetera so there's my linkedin profile please feel free to reach out for connections etc questions somebody's got to have questions

uh i'm not familiar with that one in particular

okay

okay the

yeah so so the gist of the question was that there was a statement that an employee in the government had uploaded files that later were found to be associated with with um with the solar wind situation a few months prior i don't know anything about the veracity of that particular one so i can't speak to that but what i will say is if you look at the solarwinds detail there were actually several close close misses so to speak on detection where a few other organizations saw what later was found to be solar winds but didn't realize the implications and kind of uh wrote it off as no big deal some even started investigations and then made conclusions because they just

didn't have enough data certainly i wouldn't be surprised if it was something like that also you know uh virustotal internet of itself if you're not familiar with virustotal i i it's a phenomenal resource for us as professionals there's browser plugins that will automatically upload files you know so it could be simple things like that that one of these files ended up on a computer you know if you haven't seen uh mr melson's talks on how do you correlate all of these different malware samples and derive all sorts of things from prior b-sides you should absolutely that that's your evening project because you know doing doing analysis and correlating all of this malware what i will tell you

is that's part of what i did when i was doing my investigation the tools and malware right doing cluster analysis on virustotal i again i realize not everybody has access to the virustotal back end is just incredibly powerful for helping you draw associations which can often lead to other questions in and of themselves right um other questions

oh that's a great question uh i i consume from all sorts of sources um i hope everybody heard the question because of the the the microphone i tend to lean very much towards things like your bloomberg your wall street journal a really good non-biased one that i like is stratfor you know it's a four paid analysis service wired is definitely a great resource you know um yeah there's there's no shortage of but i'm fortunate enough to to have direct access to enough intel that i tend to hear through those sources prior to the others what i will say is those are great sources for cross-referencing right so you'll find all of the normal you know

cnets you know so on and so forth will report on anything of consequence sometimes the most interesting things are the differences in their reporting a really useful thing is to go read all 10 of the articles or a few hundred in the case of of of the wikipedia article i pointed you to and you will start to notice differences and just be open and ask yourself the question why why are those differences in in that reporting could be no reason but why you know whether you're a two-year-old or an adult why is still the most powerful question out there sir ah yeah that's a great question how do i prioritize efforts certainly uh i start with things like

uniqueness right um a lot of the reporting you'll get a pretty quick hit like the ransomware is it revol is it so on and so forth right if i have a lot of experience recently in looking at events for a particular threat actor or group of threat actors then i'll probably de-prioritize it right unless there's something notable in there i also use very much uh i think the best way to describe it is uh you know a drain methodology where i take a really quick pass on the out you know just high level pass through to see if anything sparks my interest and if it does then i'll then i'll throw it in the bin and you

know this for me is a hobby not just a career and so you know saturday morning uh that might very well end up on my i'll take another pass through just a little bit deeper to see if there's anything of interest uh and of course this is where again networking is is a big deal too um you know is there anything that looks like it was novel or interesting or is there a big gap sometimes some of the things that i have end up finding the most interesting were because there was some sort of a big gap between wait a minute there was the foothold there was the results that there's a big hole in that

which might be interested in going for other questions

all right well thank you so much everybody i hope some of you take me up on my challenge and and reach out with uh with your analysis and uh and uh again we'll see you at the next event [Applause]