Wormholes: fact not science fiction - Aaron Shelmire and Christian Roylo

so the next stocks gonna be starting now with Erin and Christian and I'll let them take it from here and tell us about some wormholes cool hello I'm Erin Chow mara and Christian rural Cohen we both work at secure works we both work in the counter thread unit which is notionally to threat research team but also threat operations I'd say Christian works on the special ops team and I work in the cyber Intel Center typically what that really means for me is most of the time I spend looking at malware or specific threat actors and threat groups but over the past year there's been a lot of our time that's been spent working on a few

internet scale worms which has been really an odd occurrence in the past 10 years it really sort of all started with this in April 2017 a group of online activists or personas of sorts released a whole bunch of tools that were notionally based from the NSA in that set of tools there were a handful of exploits which were were mobile most of these were based off of SMB exploits and vulnerabilities all of those were the eternal based tools but there were also some exploits and vulnerabilities and tool sets in there that were were mobile which were not based off of SMB but for some reason they were really picked up on so those were released in April 2017

and Microsoft had previously released patches back in March 2017 so everybody patched right like we learned these lessons 10 years ago first with Ken well most recently with configure but previous to that with quite a few other worms that spread incredibly quickly across the internet well in May 2017 we found out that no a lot of people really didn't patch and there was this worm which started spreading and it was want to cry and it compromised a lot of hosts and got a lot of press play at least that we saw in the media which meant that I worked all weekend long and that what really kind of caught people's minds and really spread through

the Zeke Geist was the fact that these were based off of and and this toll was spreading based off of an exploit in the the shadow brokers release which had been released a month earlier so when we started to really dig into this story though we found that that really wasn't what had happened there was a previous version of this worm which had started spreading in April 2017 and it had actually the version number 1.0 in it that worm only used stored credentials and input and create a list of credentials that were stored in the binary itself that it was trying against different open SMB servers if we took a step back a little further we found a

previous version of the backdoor which was called W cry beta one a cry beta that version really didn't spread well at all and it looked like they were trying to spread it manually and via sphere fishes and so forth like that but taking a look at the source code brought us back to a whole set of other SMB based tools which had a North Korean Nexus in the same time frame there was a much more targeted compromise which used the same basic SMB worm technology and functions but it dropped to backdoor tools called alpha and C and Bravo and C at that organization and really that SMB worm was based off of this toast toast week

named ramble which the North Koreans had been using since May 2015 when we took a look at our data we were able to find these worms have been trying to be they were trying to spread these worms from way back in 2016 at least and nobody paid attention to it at all until it started to compromise people's sales kiosks but you know there was a lot of media impact from this there was a lot of questions about this but when we actually took our step back and looked at the organizations that we were managing their security for we had 73 unique clients who had actual detections of this worm in their environment and that's out of a bit over

4,000 different clients that we managed security for we had 399 total tickets in those 73 clients and we had 278 unique clients who requested information from us so the actual number of clients that were asking us information was much larger than the number of clients who were actually impacted when we really dug into it we also found that none of our clients had a true major impact none of their core resources or core information technology infrastructure was really impacted everything that was impacted were things like a computer over at a sales office and they had somehow they had a single internet access point that was completely open to the net things like that and then that's

also why you saw a lot of ATM machines and display kiosks that were being compromised these were all hosts that had almost no let's say core business impact and inside an organization but you know we had a lot of hype around one a cry and a lot of hype around these SMB vulnerabilities that were out there so everyone's patched right a month later this happened and we found out or it started to look like nobody had applied these patches or a lot of really large organizations had not applied these patches so on June 28th 2017 around 10:00 a.m. UTC we started to receive requests from a few organizations first based in Europe and then later based in US where they had a

massive amounts of their systems compromised and all that Christian take it from here okay thank you so both of us have been involved in a handful of not Petra investigations and of those a lot of our clients were completely hosed I thought was important for us to bring you stories about this stories from the innocent response stories to how it happened how it could have been prevented but in order to do that I need to protect names of the innocent we can obviously give client data so I've created a fictitious company protect identities of these other organizations but this fictitious company everything that I given these stories actually happen these are actual facts they're not sensation alized these actually

happened so the company that I'm presenting that I created I just call it a beach wear clothing manufacturer and retailer so presenting Seaside's that's Seaside's as in the sea you kind of have to question the credibility of a company based out of Pittsburgh that has a Sun in their logo so Seaside's they have about roughly a thousand hosts our window shop so they lost most of their host they'd already engaged a third party to start manually real building rebuilding these hosts so it's a long laborious process they called us in to help them out to determine well what actually happened how did we get impacted how do we prevent this from happening in the future so our first

goal is was well let's work backwards let's find the earliest evidence of compromise because that's important because obviously they'll need to fix that determine the entry point to the environment and then determine how it was spread within the environment because you know fixing the introduction patient zero is one thing preventing something from spreading is another that the client needs to take care of so my initial thoughts was Ms 1701 zero eternal blue because the news had come out that not petrol uses this vulnerability so I asked the client hey are you guys patched fraternal glue and say no we patched it's not eternal blue so I'm like okay we're gonna have to dig further and investigate

so starting point like I said the clients of window shop they were hosed so things as an investigator an instant responder that I am used to getting don't in it didn't exist domain controllers hosed no domain controller logs their sim was a window sim their sim hosed no data from their sim and by now the media has already reported on a few handful of companies that were completely and some of these were very large companies that not Petra affected their entire operations they were down and so I took this list and I provided to the client and client Luca listen said that one is one of her vendors and of course that got me thinking okay back to my mind I

need to consider that the burner proof is still very low so I can't say yep this is what caused it but this is something that need to focus on so where do we start well fortunately we did have some logs there far away obviously weren't Windows boxes and so we pulled firewall logs and started analyzing those working our way backwards to the initial time of the infection and the logs as we figure it out showed a lot of SMB tracking traffic indicative of not Pecha great it just proves that you know we're just further supports that yes it is not Petra you got the splash screen you got the logs we know that so didn't

really give us much more information one of the problems was visibility limitations the you know the their firewalls were placed only on certain locations through you know outgoing ingressing through outside the network VPNs they've had a ton of 24/7 VPNs to their third-party trust trusted third parties so it helped us a little but it didn't really give us the answers we need so the next step well we're gonna have to go to where the data we need and get it from there which means we're gonna have to rebuild and conduct forensics on things we think is a will contain this data things of evidentiary value so we determined we got an AV server let's do

that let's rebuild the AV server you got a sim let's rebuild the sim and possibly any early affected computers from the firewall logs so we started doing this so those of you not familiar with not Petra does to a computer it literally rewrites the MBR the Master Boot Record so when it boots up it shows that splash screen like hey you've been owned send money to blah blah blah but it also tries to encrypt the first end amount bytes of your ma or your your master file table which is your table of context to the to the drive without that you can't actually rebuild and figure out where your files are that's all gone you need to basically carve data it's a

forensic technique where we have to carve data and free or unallocated space whatever you call that but it's worth a try for us to recover the data because the were in this data so first thing we said is will rebuild the AV server it's very easy it's one workstation one drive and there's a tool we use to replace the MBR and fortunately the MFT was untouched by not petra I don't know why but through that the forensic software was able to rebuild the structure of the drive and then we were able to pull off AV logs however the AV logs was not a real benefit because AV didn't catch non pedra and stop it so there were no logs at that time there

was no signature to stop the not Petra outbreak so AV logs now been to any good however these security event logs on that host was very fruitful what you see up there make sure I press the laser pointer is a snippet all made up by the way names domains made up everything else actual names domains and target servers was a pattern we saw of authentications these were five five two event IDs so for you event ID gurus you're probably thinking what's five five two five five two is a explicit login logging on trying to log on to another resource using different credentials than what you're logged on to that box it's like but isn't that 46

48 or something it is for machines that are greater than Windows XP and 2008 or so on you're like wait a minute so that means this is a Windows XP rc1 box yes one of those that's a important point that I'm going to cover in a few other slides but basically this is logs indicative of not Petra trying to authenticate out to the rest of the environment from the AV log it's important data so I pulled this data and I mapped it like this and there were some interesting patterns I saw the domains the domain names and the accounts so one of the first things is I showed to our client do you recognize these domain names and the ones on the

bottom yes that's our domain ET net do you recognize any of the names up top I don't recognize the domains but entropy that that's a company that is our vendor that's the one that was in the news that got impacted by not petrol whoa we got more data we're like okay this makes sense now we've now have a higher burden of evidence that we know where nan Petra came from perhaps we can dig and find more then it hit me it's like hey this is interesting you see a pattern of where this moved through we did some research on some of the domains like there's a domain called EMS you found out that it was actually a

division of entropy or entropy tech called Enterprise marketing statistics and that's the other thing I didn't say is the client said yeah entropy is a trusted vendor we have a full-time connected VPN to their systems they provide a service they do statistical marketing and we use their product and of course it all makes sense now right EMS is actually a subdivision we look at looked it up on the web's called Enterprise marketing statistics East ATS is the actual name of the application that Seaside's uses okay so you can see kind of like a pattern of movement so I contacted Arin and basically said and if you look in this slide you can see the same set of user names being repeated

over and over and over again in that same path so you looked at that and that that's kind of key to determining the path of the malware we call it a flight recorder erin look at the malware and basically validated that yes the malware passes on credential is that it captures first-in first-out so you could use that order and basically build the flight of not Petra determining where it initially came from which is kind of cool because you have not just patients are on your own environment but patients are all for that chain of the outbreak other interesting things what is these names of the users searched it on the web and found their LinkedIn accounts and said

they work for entropy tech and I do blahhh there's also read you know looking at some of the names so Oksana you google that the first thing that comes up it's at you in your name we didn't talk about I'm sure you guys know the whole history about and I don't know that's something we probably should cover but the whole history of the nod Petra what has been reported in news I guess validated was originated out of the Ukraine possibly Russian government interference meddling in Ukraine it was related to the ME Doc's software which is a tax software that is very commonly used in the Ukraine it was back doored and Aaron's gonna cover that more in depth

so I don't want to really steal his thunder but looking at the pattern that first user named crane Ian so it may be Ukrainian user I don't know the other user name so we're all South Asian so it makes sense regionally how maybe this mal removed I'm starting to build this picture but we know now entropy tech is patient zero for our client so next step we're going to rebuild the symbol we want to that was a little challenging because it was a raid like a twelve drive raid 5 and and from forensics we don't want to muck around with the drives unless we have a backup so we had to forensically image the drive to make

sure we had a good drive and then we can play around and rebuild the MB are we able to rebuild the MBR MFT did exist so we could power up took a little work talk to the vendor to get their sims off we're back up and running but then we could do queries on the sim and we then identified the actual authentication into the application we call the e stats application the first authentication into the environment so now we have the server a patient zero within C sides and the authentication we then did forensics on that box so it's rebuild the MBR but the MFT was encrypted so we have to carve out event logs and then we found

the first successful malware indication via PS exec so the sim basically had domain controller logs not as lofty not as detailed so we decided to do forensics to get more information so I'm going to do a very quick nice of the graphics play-by-play of what happened so as you can see here on the left this is the entropy Tech environment there was a 24/7 trusted VPN into Seaside's Seaside's had a separate domain for the East debts application called ET net when I looked in the firewall logs the interesting thing is I did not find any SMB port 139 or port 4 5 traffic connections immediately preceding the propagation and I'm thinking that that doesn't make sense because you know the

malware traversed from entropy tech I should see a record add in the firewall so that was that was got me wondering but I did see a day prior there was a port 4 or 5 session initiated a day earlier so huh this possibly could be involved I don't know yet what I then saw is the actual propagation of the malware into the e stats applications interesting and interesting in love enough it only successfully authenticated and ran on one server at first at first exactly the raining servers within this application domain it tried to run with credentials but it failed because they would only use credentials it knew from entropy tech it didn't know when he seaside credentials

so when you combine that with the fact that there was an active session and we validated through eventually through the analysis of the malware that not petrol utilizes and abuses active sessions right so it doesn't need to actually have a credential if you have an active session between two trusted environments not Petra will be able to use that to spread so huh that's that's really interesting really cool not cool depending on what side you're on then from the host that it initially compromised it's excessively compromised other environments within the e stats environment with a different credential so now it makes sense the reason why was that different credentials if you use an active session it didn't actually

capture the credentials on the entropy tech side so for the most part w digest which is a clear text caching of credential has a clear text caching a bit ability to text sorry W digests cached his clear text passwords that was likely turned off on the entropy text side but because of that active session it was able to then infect this box our client environment because it was a older version it was 2008 not our c1 or I'm sorry 2003 there was no patch to turn off the caching of clear text passwords so not Petra was able to get an actual Seaside's credential the actual password and used to spread internally so you're thinking okay well

this is in within a separate domain it should be contained with proper security segmentations yes it should be contained within only the e stats application however what happened was there was a successful not Petrov successfully able from one of the boxes to move to the rest of the environment and within seconds basically owned their entire network since opposed so how did that happen we found that the domain admin have logged on to one of the boxes now domain admin for the entire environment logged on to one of the boxes in the e stats application because W digest clear text password caching was an able not Petra was able to capture the domain admins credentials and spread across their

entire environment this is an issue of domain trust and a poor implementation of domain trust within security so we know now how we know patient zero we know how the malware spread we know a lot about the miss configurations of security for this client that we can report to them but an interesting tidbit is like whoa who's this person who's Oxana be and it did a little web search and we found that there was a freelance site of freelance sites for people to go look for freelancers coders doing different things there was actually a profile called Oksana be of someone in the Ukraine who did very very niche work that was so related to type of work

there we go such niche work that was related to entropy text type of work now obviously this is just loose information it's just a hunch but it makes sense when you look at the path of the malware think about the story is well it started in the Ukraine again this is just this is not actual this is more of a just thinking and a hunch but it follows a story and should be Tek hires contractors hard one crane and me dog software installed on their box connected with an HP tech environment and it spread from there it slowly worked its way getting entropy Tech employees credentials slowly working its way out until I got to the

east at application and from there and moved over to our client it's pretty tricky how this thing moved but you think about it like there weren't a lot of businesses that completely got hosed by not Petra it's not like you know you're hearing like there's a good handful but maybe that's because it requires all these gates and steps to occur right this is pretty complex to happen but all these conditions existed in order for the not Petra to affect our client code so taking some of the good lessons in Christians story they're really that you should really take home from you is really to like evaluate both the network segmentation pieces but also the trust relationships you have with

those external parties frequently those can be abused in ways that you can't really war game or envision when you're first setting those up the other takeaway there was in a lot of the organizations where we saw that not petia had catastrophic ly taking them down that happened within 50 minutes in one place it was pretty quick in other places it was within an hour as well it was if it's something that I don't think anyone has been looking at or building their defenses for until this past year but when we took a look at not petia itself what we really found out was that payload the way that it was pulled down was through a Trojan I zap

Lakai Chinon the EM edocs statistics software and attacks attack submission software that update was truly pushed on June 22nd so what a lot of organizations had done was they pulled down that software tested that software made sure that it didn't cause any implement any problems with their implementation and they didn't see anything wrong with it but then the actual malicious payload was pushed six days later again on a Friday luckily which caused me to have to work all weekend again so before that they had pushed previous implementations of this back door within that Trojan high software first on May 15th and previously on April 14th but one of the biggest stories with this was afterwards

there was a lot of marketing and a lot of press about another worm called bad rabbit but bad rabbit really almost didn't impact any of our clients at all what we saw there was that were that tool had three components one of them was that this concretion component another one was really just there to shut down and restart the hosts and then the third part was a worm component but it uses hard-coded credentials for the IPC dollar share in SMB the big difference with this backdoor versus the other backdoor in the other work and the reason why this didn't seem to spread quite so quickly was it didn't pass the credentials from host to host

like not petia dead on the command line when it propagated through an SMB which meant that it didn't take credentials that were stolen from Christians computer to my computer to the next computer and move forward through all of the hosts that it could touch and those credentials could authenticate too so those were the stories that got the most press this past year but there were others worms that have been spreading and that have been being used which really had a much larger impact within our organizations in our and the organizations that we monitor the first one and the one that we see almost every single day is a motet emote it's really just a piece of

software that's there to load other software on your computer it's really opportunistic cybercrime malware but what we've been seeing is they've been deploying this SMB spreader module with it and at first it was just purely mini cats based and that was first based up and that was first introduced back in July 2017 and then it started to get a little bit more press when in November 2017 they introduced a variant of it which also used the eternal romance exploit in it the funny thing was when they added that eternal romance exploit to the worm they broke it and it only worked and it only successfully propagated within the local host that it had already compromised when it was

trying to propagate to additional hosts it actually didn't drop its payload and didn't successfully propagate the other one that we've been seeing a lot of husband the trick bought SMB spreader and this first got a lot of press back in July 2017 as well but at that point all it was doing was trying to enumerate the credentials that it could see through SMB within your organization but then at some point and I'm not exactly sure when they introduced it they also added mini cats and the ability to scrape credentials through from plaintext into that worm those two worms they've not been deploying ransomware they've just been deploying themselves and other pieces of crime work but what's been interesting

in what has been causing a lot of issues in organizations has been when those worms are spreading they're spreading so quickly and overwhelming to the main controllers and Active Directory systems within organizations that people are no longer able to authenticate to hosts and it results in a denial of service situation within these organizations and we've seen this a lot of you know in a lot of situations so how can organizations mitigate against these situations the first bit is yes you have solutely still need to patch so yeah we're not going to tell you anything here novel or new it's all fun in the nose right basic security hygiene repeated over and over and over and as

you've seen in my example if it was just basic security hygiene implemented maybe just one or two gates would have been prevented and this wouldn't have had a major impact to our client so one of the things that I do and I work on as I do a lot of thought they've been over the last four years for secure works so I looked at a lot of environments and there's almost a correlation between how ugly and bad and dirty your environment is and what's just basic stuff that that client has implemented and when we do are after a threat hunting weed or our recommendations the top recommendations are basic stuff that you've heard over the last 10 years one of them get

updated operating systems right it's 2018 not 2008 you need to run supported operating systems in the case of our client if they had to support in operating systems they could have patched the W digest clear text caching interesting enough and this is a sure you guys are aware of this right the point of contact for our client was the security guy was like yeah I've been trying to get these updated for a while for years however there's backlash because this business area needs the software and there's like they don't make it anymore or they don't want to buy the newest version and they need it and I've been told to keep it running and I can't muck around with I don't

even know like I have no idea what goes on there but I got to keep it running well this is what happens right interesting so eternal blue history of that is Microsoft actually took a highly unusual step in their own words and created a patch for unsupported the Windows XP and 2003 server so an ironic tidbit with Seaside's was even though they asked for this they were running server 2000 I'm not kidding they were running server 2000 which there was no patch for those server 2000 belonging to a specific application that they needed up and running because their business one business needed it those got infected by not Petra but not Petra was not able to

replace the MBR or encrypt the MFT so it's almost like saying security by obsolescent which is not the story I want to tell sorry it's not this the the message I want to say but it's just kind of ironic that that happened segmentation so segmentation we all know segmentation segmentation occurs on multiple levels we all familiar with network segmentation firewalls bla bla bla bla and would that help here somewhat I mean it was a trusted VPN connected that had SMB traffic already open right so you know it's Network segmentation may have not had worked properly there for this case the client had an open flat network and there were other sub there were other clients that

shared the network other organizations that shared that network that were also impacted so segmentation definitely would have helped but segmentation goes beyond work domain segmentation you know segmenting your domains and from enforced and something called privileged access hardening which Erin is going to cover next yeah so what Christian was saying with network with segmentation is that in a lot of organizations I think they think that once they implement a firewall and set up that sort of policy between different parts of the network that their network has been segmented and what we've been seeing time and again at the top of this large infographic of sorts is an attacker comes in they just compromised a user

workstation from that user workstation it's almost like clockwork we always see this where they just use mini cats they dump the clear text credentials from their the House of hashes that they can get access to and then they pass the hash from host to host a host that allows them to access the domain controller eventually because at some point a domain admin has logged into one of those hosts that they compromised they take that domain admins credential and they're able to use that to access every single host in the environment we've seen it in these automated worms like not patio we've also been seeing it in more targeted ransomware cases like the Sam Sam cases which have taken down

other entire networks and then been ransom one-off Microsoft has introduced a new set of technology and I don't know how new this is anymore but it came with Windows 10 in essence and what that does is and and their recommendations are for you to use a privileged access workstation and that's the little red computer up in the corner and that privileged access workstation allows only that admin to create credentials on that host and then if that admin needs to have actual can only have one laptop or one workstation they have a virtual machine which they'll use for their actual user land life to access their email to go onto the web to look at sharepoint or whatever now taking an

even larger step from that is to create this and it's truly a four tiered set of credentials but here it only shows the first three tiers which are based on your administrator credentials a tier 0 your domain admin will only live within tier 0 and only authenticated tier 0 that domain admin credential is not usable and not useful for accessing the next tier tier 1 is made up of your core workstations and your what's the term I'm looking for your your core workstations and your core and your core servers that put business-critical applications out there and that's separate from your domain controllers in your Active Directory administration so the credentials from tier zero which are

Active Directory administrator credentials can't authenticate and can't administrate your work your your business critical servers the Tier one credentials have to stay on tier one they only access and they only authenticate to your business critical servers but then those credentials can't even access the workstations that your normal people are using and that's what makes up tier two tier two are your administrator credentials for your for your helpdesk and for your user workstation administrator staff those credentials stay there now in this and then the fourth tier which isn't shown in this image is made up of your normal user credentials using this model whenever an attacker compromises one of those workstations through a phishing email which happens every day everywhere

when they try to scrape those credentials from memory they're no longer able to use those for anything other than accessing other people's workstations and other things on that tier if you're able to keep these things clear and clean like that so we strongly recommend that people move to this sort of model and that level of segmentation in almost all of the cases where we've been seeing entire organizations compromised and and put in terrible situations with ransomware but then previously with a lot of the apt style intrusions it was because there was usually a single admin which allowed you to authenticate and administrate all of the hosts and it was that shared admin account which the helpdesk used the

server administrators used as well as the domain administrators used and they had a single password equally or you know a single set of credentials so once you get that hash to pass the hash across the organization so in summary the mitigations our patch absolutely we put those patches into place and this is pretty basic and pretty standard stuff we're not selling any special things but if you want to buy stuff you can buy stuff from us too but also segmentation which is way more than just firewalls and really includes your different separate Active Directory forst's but creating that privileged access workstation and tiered model really helps organizations improve your security posture plus disabling wmic where possible the W digests credentials

and scraping those really is something that we see all the time there's there are a lot of single sign-on applications which require W digests to be running and actually require people to use that we understand that try to segment those organizations and those parts of your organization even more so than the rest of the organization yeah and if not privileged access workstations don't use a single account to administrate all of your hosts that is our presentation we've got four minutes I think [Applause]