A Deep Look into a Chinese Advanced Attack

so anyway thanks for coming to the talk I hoped you one of the comments had gotten yesterday from my training was that was really good actionable information so I'm a big advocate for talks that have actionable information meaning whatever we teach whatever I teach at least we should definitely make it a focus and a objective so that you can leave here going I can actually go back to work and do that and that's really my goal so with this talk unfortunately in the course of my last employer I had already accepted an offer before this occurred last Thanksgiving and so I did stay with them all through the point of getting this contained and beginning the cleanup or at least we had

it completely isolated so we could start to clean up and then it became my buddies problem but I was there long enough to document everything that occurred and so I'm going to share that with you in regards to how a tell you little bit about the Chinese apt that we're talking about but also how we went about finding it now for those who don't know I do the logging training and yes I am a blue team defender ninja and I'm our archaeologist I really had to talk to me about that later and yes my name is Michael I'm a log aholic you're better than my training class was so you know I love properly configured logs I

teach a lot of people about logs in and tell us who they tell us what they tell us where when and hopefully how obviously the why we can you know discuss that debate that all day long we got an interruption to feed okay there you go and i think i'm the creator the windows logging cheat sheet that's a real important component to this talk so it's a resource for you to take away you can google it and or just go to Mauer archaeology come and get it I am coming out with a Windows PowerShell logging cheat sheet thanks to ben 10 though that some of his little hackery of launching powershell without using powershell THC

I gotta go update it so thank you ben 10 my my internet internet con son and then i also recently it's lung comp it's a long story dallas dfw i also just released its blunt conf the windows splunk logging cheat sheet I got a lot of requests because I talked about using splunk in the course of many and people I've worked with and they say well you've got some short list of things that start me off so I can kind of get an understanding that's what this is for this is about six pages I think about five pages and then a huge component of this talk in regards to understanding how to start looking for advanced

attacks is what I call them our management framework the idea here is everybody knows that vulnerability management is we get alerts and notices and vendors we read some of the vendor reports we possibly take action and eventually patch the system so it's something that generated from a vendor the information in it causes us to take an action Mauer management's the same thing I read the apt reports from eyesight partners that brought up wrote up targets breach in January I then make a lot of fun of home depot who got breached in July later that year because everything they needed to know was in that January report and I read those reports and I pull the artifacts out

thus now our archaeology we collect artifacts of ancient mankind malware's written by man now artifacts right and basically you take this information and you can pilot you start improving your active defense am I looking for this thing in this place that this apt did am I looking for this artifact right some people call them ioc's a lot of people think I o Caesar md5 so I don't use ioc's for that reason but now or management framework is when I want to tweak tools I read these reports I see that they're starting let's say use the windows font directory and so I'm going to make sure I monitor the windows font directory for creation of new or drops

of new files doesn't matter what type for example and this is really beneficial on the website Maori ecology calm there is a link of resources where does a whole lot of these I've collected and so if you want to start that concept of reading those and get to understand stuff highly recommend it i'm also the co-creator of log emd it was a tool we discussed yesterday in training i'm actually going to show you a little bit about that the end just couple screenshots the idea was i infect malware in a lab or i have to rebuild them our like from this attack into a lab make sure i know everything is doing so when it comes

time to pull the switch and get rid of the malware and bandra mediate the problem I know everything I know and that gives us high confidence level I needed a faster way to gather some of that data and so Brian Boettcher of breaking down security and I basically wrote this tool dumps out a spreadsheet of all the data within the logs it's going to do some more and it won't run until you actually configure the logs based on the windows logging cheat sheet so there's a tie-in between the tool and cheat sheet and it will drop you out a nice audit report that lets you see what you need to set you would actually go

through and set all these things when you finally do the tool run so with that what is the goal interaction there are some things in here you probably want to ask a question on so if it's a short something I can shortly respond to don't be a ding dong and not ask questions the only stupid questions are unasked ones if you ask a good question and it really quite appropriate to enhancing the content I will Sam well because now I lost a hand send you out a ding dong to encourage you to ask questions so don't be a ding dong and again learn how it's ninjas do it so you can do it too and

I'm going to expose you to new tool which really came out of the fact of I need to catch more of this stuff so it's that so with that why listen to me well how many people here are familiar with the gaming environment MMOs likely legend stuff like that that industry has been attacked by a group called winnt I for probably about five years that's what they label them kaspersky label these guys that's where that comes from and this report came out april 2013 right there but we found this malware in 2012 and met with the feds share them that information so you get declassified and shared with other people a good chance like especially got a lot of the

information that they needed for the support from us but they didn't contact us I kind of ripped Eugene about that one because they missed some things they really need to be in that report and the other thing is the talk I'm about to tell you about we gave an infected vm to one of the big I our firms and no I'm not gonna tell you who because it doesn't matter everybody has a bad day or three or week or however long it took them to fail in this one but because I do respect the firm you a lot of good work but the point here is you must validate or at least have some understanding of

what you're giving these I are firms if you're doing what we did give them an image and say hey what do you know figure out what you know about it first so when they come back with a report it aligns with what you think and hopefully expands on what you know so that's just the point there so yeah we find it before they report do the reports and we deal with a lot of our firms and the bureau to share and declassify information so when NTI 2014 it was a much more sophisticated version of this particular kaspersky attacked a guy they updated their approach quite a bit for example 2012 used new services much like

the black POS did at Target and all the other retailers installed two services the win NT I guys installed lots of services that are different across a lot of systems they didn't do that this time so suddenly they now know we know to find new services so they had to change their mo as we have all they evolved and so are we really have to up our game and suck a lot less but boy did we catch them in the act this is definitely not your typical pwnage I hope to really get you guys to your eyes open with this information but bullying the whole 210 days mean time to detection it's more like 200 10 minutes for where we're at I

can pretty much get something going within that first hour if those systems are monitored and it's a gaming environment every system was monitored so we caught them right when they started again they can infect an entire environment within an hour to wear as it takes us an hour to just to evaluate one system right so their way faster than we are and the goal here is I want to share this so you can learn okay clearly we have people in DC that are very interested in this kind of detection I'm hoping or maybe avoidance of this kind of detection it all depends on your perspective so the history again I mentioned it's been around for about

five years it's known that their Chinese hackers both kaspersky obviously mentions this in the report so let's believe them but also the feds told us that they're a known group that's being tracked but again they're in China they attack the gaming industry and everybody's like why would you do that it's considered a billion-dollar black market industry meaning when you farm for things and collect things and they steal your things log in your accounts or create new accounts and script collecting of stuff and they drop that off and sell it on on ebay for example I think before I left our gold was going for 100 gold was going for like nine dollars and so there's real money behind fake gold and

fake things surprisingly and and so they're very much incentivized to rip you off and always the sunday so this is a very organized group matter of fact it's known that they have a building and different floors do different things many of the gaming people have gone over there have been told by the authorities you know there it is you know that floor probably logs in or creates accounts and scripts gaming these guys break into accounts these guys write code highly organized so and again it's not quite state-sponsored for sure there's some good stuff in here but it's pretty darn good I have to say this the best stuff I've ever had to deal with and i work

for HP for eight years and got to expose us some interesting stuff and i would definitely consider this your typical advanced persistent threat now advanced i hear a lot of terms with malware where this is sophisticated malware like I heard that with all the black POS stuff I'm sorry that was one of the easiest mauers ever to detect so it is not sophisticated sophisticated to me is when something new occurs that you weren't looking for thinking about and this stuff had it in there I'm going to walk you through it and again we saw each new things each time the attack could obviously keep learning keep earning disease these persistent malware ians or bad actors or whatever you want

to call them keep coming at us it was almost like a monthly game with us you know they specifically target HR with really well-crafted resumes that were infected because a gaming company just laid off 5,000 people and they know at that point it's time to release a resume and be amongst all the other resumes that the HR person just got my colleague had an interesting thought maybe this is state-sponsored stuff getting a test drive and the stuff that doesn't get detected maybe then gets elevated up I'm interesting theory but I got no backing for that but that was an interesting thought that maybe that is kind of maybe the Chinese or even the Russians would

buy this good technology that took three years of detect yes right kind of this athlete so here's the summary of it pretty typical from 2012 DLL injection they dropped a dll in wbm that looks just like named exactly like one in system32 they dropped down a windows so the idea here is in wbm when W Mike starts up in stocks and million times a Sunday on a server and in many workstations the box got infected but nothing more than adding a dll to the box they dropped a dll into the windows directory solely so when someone logged in and infected as well alright pretty typical they dropped files in system32 because they could obviously if they're

dropping files in the other two directories their admins they also use the program data directory where they stored files why because it's the all users directory right everybody who logs in the box has access to that area a well-known exploit sysprep calling base dll was executed and used quite often so if you write a bad trip based dll and you just tell sysprep loaded guess what it loads it this is known I've actually seen the expanding of this with executables and some commodity malware I had matter of fact one of the talks yesterday with a 10 dot exe my talk I the nine dot exe actually utilized the sysprep piping it a bunch of executable

so I have to write that up and so into Microsoft because I think there's something now and exploit kits that's really expanding on that so watch for the execution of sips prep is probably the best thing to tell you there I did a boot up back door that was really cool they put it on our exchange servers it actually wrote to disk on shut down after tripwire stopped and boot it up and loaded before tripwire stop started so trip are never saw this file on the only way we found it is you know when you're in this kind of attack scenario eventually get around to doing memory dumps and there where it was we had

black hold all the ip's involved to a rackspace server running netcat just to be able to keep the communication going and about every 30 something days this thing would open up in phone home or try to phone home to get a connection back it was their super-secret back door but it was pretty easy to find pointed to a file on the system 32 just wasn't there pretty easy but we couldn't figure out how to get it off disk it was real pain yes they datacenters in taiwan and amsterdam dallas and it's really hard to take full images and copy one with the wire and we tried a bunch other things you know quick we're trying to figure

out a quicker way to do this memory dump by process was the fastest way and turned out one of our offices lost power I mean the batteries died and everything brought everything back up and I'm watching these servers really carefully trying to figure out in the course like couple months after this attack you know what is it that this thing's how can we find this thing some of the way the memory dumps and the servers no longer talking the remnants are all gone the memory dumps nothing I'm like what happened sometimes a server so I start Splunk in it and I find the server died rebooted and it went down it was down for a long time and I'm like what the

hell oh we had a power outage it like lost everything UPS's were overloaded and they killed I'm like oh yeah I memory only stuffed full of power so guess what I did walked in the server room fold these exchange server power and boof that was going to it was our backup it was the one we had retired and boom it was gone I'm like awesome guess what we have is a new process every time we find when these infections power down the paps in the databases pull the power and boom so it's become part of my incident response to do that FY you really should too yeah I just do it before you shut you shut the database

down first that would really really bad idea and also they install a lot of new services a multiple of different names Splunk drv sis all right so splunk dear views of service and we also right above it head splunk that's like wait a minute and also multiple infections per machine so they didn't affect a box the same way once they would in fact let's say in this particular case are probably 200 systems they would infect 200 systems probably eight or nine different ways one box would have three another box would had to another box would have something totally different and they kind of littered things around hoping if you found these sort of things you'd

miss these others but they were all having the same logic so pretty easy to find really so 2014 new stuff dude we were is like dude you know I'm an hour guy i'm an active fender so when i say really cool foo i'm just like dude this is awesome it's bad but it's really awesome because hey I don't have a talk yet it has some more done but again what triggered it what changed you know it avoided the message method used before not a new service and fortunately we're doing really good logging I I would have to say if you higher rate all the security solutions member is work for HP I have seen and played with a lot of

security solutions my number one solution is some sort of good logging proper logging number two is my case not because I like IBM I don't work for a vendor is big fix I've used it multiple organizations and that is absolutely the baddest-ass tool the best description of how i use it is like mozilla investigator or google adwords fonts i ask a question and all the systems have to reply back with an answer do you have this file in this location with this hash rollin and I can do an analysis in big fix that gives it to me much like Meg would do or girl would do and again we we are ninjas after all at least we think you very and

so here's the summary of improvements let's walk through these plug X was heavily used initial infection plug X as an exploit kit it's very modular and so they can start off with that and then write their own modules with more stealthiness does a lot of automatic bypass a baby and various other tools as part of the exploit kit so we saw that initially used for the initial infections as well as some of the litter different pieces across the environment this was new they actually did a DLL injection on sequel server on single servers like you know program files microsoft single server ms underscore sequel 10 nada slash bin and in there was this robe dll and we're like what

the hell is this thing doing here and so that was one of the last pieces we kind of we kind of went after but it ended up injecting to the point where they could create they could get in and compromise the sequel server to enable XV command show which allowed them to do the dotnet commands within the sequel server which was totally hidden because they know most of us don't log sequel server to our security logs so in that aspect they could go unnoticed in this Avenue there is a way to do that so I highly recommend you get your your security sequel stuff into the base windows security logs because I don't have to

change splunk it we just automatic collected there are articles about how to do that so that was the takeaway from from that particular item they had a binary infector this was the dude so so we ran McAfee for example AV we had backup software and imaging software from altiris bigfix obviously several other components long etc they actually had an infected infected our management software and the management software still worked and we'll get to why and how that worked in a minute they had a driver infector again that the way I catch these things with this really cool high-tech script robocopy everything in this directory to captured only try once and skip these other directories so I don't get these

funny symlink loops and it just goes next it goes back the top and just does this over and over and over again so we started launching that across their environment and so as the boxes got popped and these things run you don't know how malware works they drop a file they execute a file delete a file with tripwire you have about strip bar does kind of like a 10 second sampling so if you can execute something within 7 seconds and delete it trip I will never see it okay FYI so they know that and so Mauer generally does is very quickly the only way to capture it is to try to get it while it's doing its thing so I have

this high-tech method robocopy everything to captured and I run this on my lab boxes and I will deploy this as a part of IR if I think they're actively unboxed or will clean a box so they come back and nail that box because we know they like domain controllers and exchange servers and we caught these infector 'he's what's really cool because they need to figure out what they did and how they did it because right now we know is we have infected exe and infect our sister you see they hid stuff in the registry they hid their scripts that they executed in the registry and they also hit hit their payload the registry and folks the

registry is seeking I do good Donald Trump the registry is huge and it's a big database so hiding stuff it can go totally unnoticed and surprisingly there's a lot of stuff that can be hidden in the registry so initially they popped a user it was not an admin so a they're good we're pretty sure or at least I'm pretty sure patient zero was an office oday exploit that had not been patched yet because afterwards and we got some patching going the next patch cycle some of the stuff we tried didn't work so we're pretty sure that word doc or thing that that user got sent he was immediately imaged and we don't generally do that but all the people got

infected because it was a fast rapid roll he was on vacation his went ahead and did a bunch of systems I don't really care at that point I have a couple systems and I have all the data but generally you try to keep that patient zero because normally in these attacks patient zero looks nothing like patient one or patient to a patient three through whatever generally looked the same in some aspects they drop their initial malware they use the backup software they harvest at the backup software creds and hopped on a domain controller and from there blew across the environment and they spread all over say memos 2012 they always went after the stupid domain controllers obviously

that's where all the creds are Mimi cats or whatever their equivalent is they run and easily pull creds off now in this case a 20 2014 I'm pretty convinced up whatever they're installing on these boxes actually needs an admin to log in before they can steal the crits because that was a pretty much almost a you know hey Bob just logged in the thing I guess what Bob at 10 15 20 minutes or an hour later just popped through my boxes so I don't think they were harvesting right out of the domain controller I think as people logged in they grab their creds so same MO that aspect the same behavior in highly crawled and so files dropped

and gone right they used publicly accessible directories to drop this stuff why because they didn't worry about user they were they would just drop it we're all users had rights so if you're not monitoring see users public or see windows web or c / flogs you better start doing it not to mention the whole users app data local app data local oh updated roaming directories and then they deleted the files fast i mean we're talking less than a second we could watch them booked on but they did leave someone disk obviously the persistence must remain in some way for manner and so they were gone definitely in 60 seconds the sequel server being five deep they dropped a dll called

cease capping it is a dll that's required by sequel server so if i put the bad one right next to the executable the a windows you will load the bad one first before you go to system 32 or wherever and load the kid one yay they drop files in 65 64 qaf dll is normal on workstations for sound stuff it is not on servers so they pop they use this thing across the servers as one of their mechanisms thinking that a QA is normal not on servers is not socially VMS they went after the splunk and out terrorists directory where they dropped a driver why a dot sis or a driver windows doesn't log this stuff very well if you

want to log driver load you're going to need you something like sis mon or the windows logging service and so they named it to look just like there is no spunk sis in the splunk directory by the way spunky ll yes punks this no same with Altera's so these are the same payloads and so they litter these things across the environment looking like our management software that was a big thing they did this time the initial infect our again dropping to see users public because again that's everybody when they log in they can pull this they had an item called CXC infected exe infects sis so this is a thing that infected the binaries this is the thing that did the

drivers and then 64-bit obviously the dll for that wasn't really sure what exactly this one did and then one was specifically named after our company so that means there's probably a blizzard exe and a you know whatever riyad exe is well probably out there how many people here do logging good enough that you would see this execute on your systems okay I see one two three hands they don't have to the bottom line they don't have to they do in our environment but their writings for the entire gaming industry probably owe their d leave within one second we wrote an executable that wrote filed 0 seconds delete 10 times wrote one second delete 10 times

in order to figure out where the threshold was and yeah we watched him we actually recreated this in the lab and it actually was able to record how fast it was yeah won't be there long / flogs had the command binary stored there and then windows web had a bunch of any files for permission changes the temp directory and windows had the vb scripts and the ax script and since well 64 had the files dropped this is the dll that was obviously a show she with that particular infector this guy here and that guy is a different piece of malware this guy here is the same as the Alturas and the splunk sis same same thing all

the sis files were the same payload just littered along the disk in different ways and infecting different ways so they made it they made a they pointed an actual new service in this particular case when they use this one it was a new service on a couple boxes thinking oh we caught this and they wouldn't see the other one that I'm about to talk about so perf log co XE yay for once we got there creds so if you look at this whoops you look at this slide here / flogged see exceeds being executing and here's the command line obviously port 445 and we caught their password so now we had the ability to actually go to their command and control

server within our environment and see what it did so hey winning thanks for the port and password for once we compromised them so now who sophisticated persistence was really cool in this case they created a driver in this case this driver again same payload of spunk sis and I'll terrorist sis it but in a different location but they put it they pointed it at program files common files you know with W x64 sis but it wasn't our disk anywhere couldn't find it it's like why would they modify the service dll in this particular by the way it was the worst CPL support so who needs windows error reporting when they're infecting your box so I didn't want to use it they

didn't want us to see it not that anybody monitors anyway so he pointed this thing here and we're like wow why what goods that do there's no file and we're like okay we know what's going on here something we're missing and so this is what it normally looks like right here so they modify that injury and it turns out that this is what the entry looks like when after they messed with it so it actually isn't that it's pointing to some oxy ated code right in some cases mistakes they left it as an MZ in some cases it was put it in hex as a 45 a but in all the successful infections that had been encrypted in

some way and so all we knew was it was a big payload so I did it load if it was not on disk well yay windows support Cpl failed to start yes thank you microsoft flaw number 5 million 425 when a service fails to start because of a missing binary or whatever they point to it loops it doesn't try once twice three times a lady it just keeps looping and looping and looping which means you got lots of time to get that file there right that's that's the key there and yay no thank you there but once the file existed where CPL support started and the system was now infected and then the file immediately disappeared like what

the frank we still hadn't found the file and we know because it was a cyst file we were pretty confident was probably the same binary that was our assumption but don't ever make assumptions other than help you go a direction when you get stuck i'll let the data do the speaking so when we actually hash the file they were at the end of the same file all the drivers seem to be the same so again persistence after leaving the key falls behind I'd like to do before well at least one anyway she uses public 64 SS started telling us some information I'll tell you how I caught this here in a second and then of course a reference in

the windows so you asked how I caught some of the stuff this is how I did now if you haven't figured it out by now this is command line logging that now is built into windows 8 dot one and twenty twelve server r2 if you tweak a key it is not on windows 7 and 2008 server at this time last year in February 2015 of this year Microsoft added a patch it's kb 343 75 if you do not have that patch i highly recommend you go put that patch on there you tweak the registry key and you start collecting this with a process create 4688 event so those servers that i had been playing with this up to this

point one of my summer projects was to get this command line long and going so a couple win 81 boxes we had and a few new all the new servers for a new game we were doing we're all 2012 servers i started turning this on to see what we do and i started building spunk stuff i hadn't quite refined it yet but the cool thing was anytime command line was executed if i went net use whack whack some server wax some user whack / password whack it's in the logs I don't worry about having passwords in logs so I tell the developer hey take the password out and do it through a file or something so we don't see this or I can

tell splunk not to even collect this particular string which is even better way to do it so I don't consider it a problem collecting command-line logs I can remediate that problem and so I started collecting this information as we start looking the boxes that were infected and it's pointing me to locations that I now have to monitor or look for across the board using a big fix analysis hey do any use servers or any boxes have 64 tied yellow in this location and boom I get its response banging about 15 minutes do any of you have this file here the script in the temp directory etc etc but the other thing it did is point us to software

clients like good man that's a reg key what the hell and so we went looking and basically we started seeing some interesting in the registry in this case the command line logging was the key and I just told you about this and we have a big fixed thing that looks for several a whole list of Edmond utilities if if you want to look in the windows splunk logging cheat sheet you'll get a short list of a lot of them there's several more I have it was just I was trying to make it short enough and basically when that thing gets triggers we get an alert because of the multiple executions of that yes oh you haven't problem sorry

but then if I'm doing that just somebody do this wait your hand and go I'm sorry I'm trying to hold it against my body so I keep that going and so what we found is and this is where hackers don't look anything like Edmonds is an admin will go to box type net use maybe run see script on an exchange box to clean something up a it's always an exchange script be it's always this script or powershell script and see he's only doing it a couple commands but in the hackers case they executed see script they executed stuff about the registry the executed net stops they executed command XE with C script they executed a

push d which is a drooping changes a directory not a heavily used command in Windows but it's there they also did take own because now they're trying to take ownership and hide the dll for from us they hit them at rip them and all that and so right there the quantity I've got just from this one infection allows me to set up a logic of if all the if all these executables in my list execute and are greater than quantity say to that means on the third one of these my slump would alert me to multiple commands being executed in a box I rarely caught admins doing this unless they were installing a new system

and I can just reach out and say Hayes issue and so when they started doing this in a box as we had command line logging it lit up like a Christmas tree it was like win win win win win yes hey Tori they're moving forward it's already there yeah I and on all the windows stuff now has this option you just have to tweak a reg key and so it's in the windows logging cheat sheet the reg key at the end we get a didn't have anything go and incoming so hidden the registry a via the command line pointed us Hey look here software clients there was also a software classes key they used and the classes asterisk even

better yet really trying to hide it not making it a name and inside this was a bunch of info one of which was in this funny scenario of this big blob we're like hmm what's up with that right now I still have no answers they pointed me where look but I can't read anything so you know we kept poking around kept poking around kept poking around in some cases especially when we started recreating this in the lab because at different stages they literally copied the driver out of the file system and dropped it into the registry so it was MZ and all the normal code you normally see and in the course of their infection

they would convert it to hex as part of their office keishon and then from there they encrypted it so when we did see these in MZ we are harvest them when we saw him is 45 days we could harvest them and turn them into MZ's we could hash them and now we knew what was hiding in the registry and we didn't care about the encrypted pelo at this point we knew it was that driver and so that led us to the fact that in this case here's the here's the big long script that sort of looks like when you're looking at it I don't know what that means and so you have to start trying to get clever and

figure out how to harvest it now this was new they've never seen this before and it led us to actually changing some of our behavior and doing I are they added three new keys to the registry they put him in the software clients or classes key again trying to hide in multiple ways and they also on the HR system created a what looked like a w acrobat file it was the same driver we don't believe this instance actually successfully installed it failed it's a good thing because this person was infected months early but it definitely hung it was up something it did not work in their environment they created three keys in English put file file and read and

here's what was in them no DXE that's not a typo that's what they named the key who knows yeah but in there was the driver and for sure it was an MZ as a PE windows fifo what's that yeah I love Adobe apparently but yeah that's the name of it that's you can't go off a name yeah true so in the foot file directory they had this big blob again you know there's not much there but that's you know actually the the stuff they were doing and then in here you can actually see one of the ones that were it was done in hex so this is the one I harvested to figure out what it did

because if it's in hex I know I can make it actually into a PE and I was able to hash it and see that it was indeed the same drivers everything else so I now knew what the thing was that was impacting the box the same as all the others and then and again if you don't know 45 a means m ZX whoops and then they had this read key so in essence what they did is they had the infect or reach out into the registry after it was populated grab this registry crees string which was the actual script information to infect another box then in order to do the D infection or they or they pull out of the registry and

write to the disk they read are they did the foot file where it actually told them how to pull it out of the registry was all the sea script and whatnot that then wrote it to disk and so their scripts are in here again obfuscated it but in some cases not and then the big ass file payload was a 296 Meg file payload of a driver and I'll talk about that in a second so the infect her again was for the DLLs to infect mcafee framework frame mcafee framework service and make me framework helper sorry let's make sure I get the right one and the infect sis was a thing that did the driver infection so we actually use

these tools in the lab and said yep that's what it does and it infected the mcafee framework service the best client helper and several others and some they failed I think by accident or maybe this tool doesn't work on everything but we tried it with a lot of others and we were able to infect and run no pad for example it was surprising this vector works really really well and yep yeah it worked and so now we could recreate the payload in the lab which means we knew everything we needed to move forward but one thing how did it get from the registry to the dropped file location and well here it is there's the location again once we

executed the bower and saw that mcafee framework service was doing it and check the hashes against all the others we found the systems that we had the initial artifacts on they all had these different hashes for these different management utilities and we're like aha they popped them but they still worked AV still work big fix still work alterra stuff works Blanc still work and like what the hell this is this is really cool and so that came up with a dude so as soon as it loaded it deleted so now the now that the comp recover mines binary loaded read the key decrypted it dropped it to disk now this looping service could read that file load it

then the things that okay you're done delete it so that's the way they tried to keep the persistence where we couldn't find the file and of course we were better than they were so yay they worked really well so what led us there Mallard discovery I talked about the concept of baselining in order to figure out where these binaries were I said something on the disk is different so we immediately initiated we I have this big master hash at windows 7 windows 8 I had to add to it and windows 2012 already had from all the other servers were built and windows 2000 e I had run a baseline hash on our new builds so I had

his big ass file and I threw that in comparison with sha-1 deep with a minus x2 one of these infected systems I just made a script of big fix and push down and said run this drop the files that i went and harvested them and what I found was in the bigfix directory in the mag raphy director emailed terrorists directory is one binary had a different hash than all the others and that point island I'm going to go look across the environment and hash maxi framework service for everything and then I must sort it and I see hey all these have a different binary and all that spigot there's a stud bits there's those tidbits and that's how I went across the

environment and found them and so now I knew exactly which machines are infected with that particular mechanism that was good and you do to compare like plus a master file or paws am fr and so I did it compare and it showed us listing of hashes and that's how that's how we basically do it if you're an IR guy and you've ever done this Sean D but this recursive hash stuff works pretty well we looked about them and again oh now we're winning for sure because now we know exactly the missing piece of what to do to clean it up alright but I want to see it do it so everybody's really proc mountains internals tool for

windows well there it is actually doing it so it's the best client help are doing a create file c program files common files system wls 64 cysts and there it is actually writing the file disk that was the final absolute we're certain that's what it's doing so pretty cool and then further down you can actually see where it deletes it BAM got them man we now know what we need to know so finally we had all the pieces recreated in the lab high confidence for remediation now able and this is about mid December and started just before Thanksgiving and it did not take 210 days meantime to detection this was literally within first couple hours

contained within the first couple days and and basically at that point remediating you can't just turn off game blades in the middle you have to do maintenance windows and all that crap there's nothing of value that they can take on those so didn't really bothers too much they were contained they couldn't phone home but once we have everything we turned off all the communication there were certain we had everything we needed except for the fact that one of the blades unfortunately had crashed before the cleanup and so when the guy found it crashed and brought it back up reinfection so warning be sure to work with your guys make sure all the drivers of all

the management apps are running before you start your clean up an account for every damn system because you skip one yeah yo start all over fortunately the same stuff so we able to immediately cut off communication again and start over and this time we went through a lot of validation and make sure everything was running before we cut everything off so yeah it worked pretty well so how long was it from the first got into a clean up to begin a remediation so we detected it within the first couple hours we remediated yeah good good question we remediated we contained it in the first few days we cleaned up Enterprise probably in a few days later and until

them course that box yeah that box came up but we had a lot of other components we could not clean we're talking hundreds of frigging game servers and all the components all the others help rebel servers and and because they're internet-facing there's no firewall controls like you would think in normal environments because of the throughput we didn't have the same ability so when we cut stuff off we had to do it you know all edits one at once because of all digital data centers across the world so that was scheduled they weren't able to talk to any database servers so we don't worry about risk of that stuff it's more of them just trying to hide

all the persistence and being able to hop around and go looking and getting all the stuff they actually want gaming has a pretty well design infrastructure and so there's nothing of value here let them have it it's not crashing the game don't worry about it if it crashes the game rebuild a server push it to another one it's a good question it probably took about two tries of two weeks to get all the things cleaned up the rest of the stuff that was contained in the bubble so now our management allowed us to set up the alerts because I'm constantly read these reports I recommend you do too I take the tidbits him looking at the fonts directory I

pick on that one because these guys use the fonts directory to store their log files their debug log files I this ran yep successful yep successful and it was a very detailed it was awesome so they forgot to delete them but I wouldn't found that if i wasn't monitoring the fonts directory which i actually got from another apt that's how ma'am our management works now our discovery because I'm hash into boxes saying there's something on here i'm not seeing or i'm not looking for in a directory they're using that we're not managing or monitoring for so i'll get a hash the whole box that's what my our discovery is involved and then malware analysis and doing it recreating in the lab gave

us all the sure ensuite knew we could remediate very fast so how you can detect this ninja tips this is how we harvested the malware so some infections hung so recommendation number one is look for parent list executables so for parentless processes for example in the talk yesterday and also my training we reference on my slides a malicious word document in that word document when you launch it there's embedded macros in vbscript so you have a tree in the process tree looking like this it's windward its command exe its see script it's powershell it then calls another command a txt and some other executables and so you get this tree structure you close word command exe is still there

but now its parents gone so I can crawl through systems or the simple scripts and sis internal tools will do this for you and basically final the pigs running and then say who's your parent because there is a process structure within the windows and you can identify that and show me all the parent lyst identifiers so when the stuff hung in the box yeah the parent was gone because the effector launched assuming it would complete and delete it when they hung I was able to go harvest I knew those servers had something on the box and I went harvested it so that's a big tip that this was really valuable for us look for parentless processes there's not a lot

of them normally some stuff like Java will do it but there's not a lot normally the other thing is the command line logging obviously pointed us where to look so we created these really cool high-tech scripts with the robocopy here it is nothing more than woohoo I changed it just a better for prezzo but was captured I used the word captured but I slewed the directory capture I want to capture my captured stuff over and over and literally just copies everything loops loops loops we turn this on on all those directories I talked about earlier and the cool thing was as those things ran and when that second infection happened after we were doing cleanup so

as we did clean up one of our steps was to launch these scripts and just let them loop and raise directors are pretty much empty there's not very much stuff in where they put this stuff and so that these things are I'm really fast nobody even notice they were running and so when we got reading that we got so many binaries out of this was great yeah simple high tech right but it worked awesome if you're in a lab environment where you're taking payloads and you're infecting them highly recommend you do this where you think the mayor will drop obviously it won't catch don't do entire C Drive that won't work but directories they don't have a

lot of stuff this works really well and anything in the app data space for all that commodity user base drop malware is awesome so top priorities here's the kb guys everybody needs his patch read this it also tells you about the registry key to tweak I you know I call this some people mentioned I see my sexy six talk this is kind of an elaboration on that especially because of my sexy six and that was built to pick on home depot and target target obviously got popped we know that the isight partners report showed in much detail that here's what it did one of the things that did was dropped two services black POS d XE and

POS WDS if all these retailers home depot especially had monitored for nothing more than in this is XP these are win7 and on vista I'd beat IDs but there's an equivalent if they had monitored nothing more than a new service install on a POS they would have caught this thing immediately if that's all they looked for that's a that's a huge payoff here folks if you're not looking for these two items right here this is your short list of stuff process create get stress on it's noisy they'll get me wrong read the windows login cheat sheet to get some guidance but if you turn this on now everything that executes sin the system ever recorded in

their logs and the windows logs 4624 successful logins of course 51 40s this is where I'm capturing them net using two boxes and pushing their payload which they then executed remotely with wmi 4663 s I have auditing on and file locations like app data local see windows web see users public and see / flogs because nothing ever happens there it's not noisy doesn't tilt the logs so that got turned on and because now our management tells me these locations are often used I do some testing how much is this really putting in my log normally that's a good alert and I'll leave it there and now I can catch file drops just from logging 5156 if you don't use

the Windows Firewall another talk yesterday talked about this was huge value for them sis mom will also do that you want to use a system on service we did use that as part of the IR here put it across a lot of servers the windows 7 2008 servers it stopped a lot of times so it wasn't as consistent as I'd like apparently Carlos Perez is talked to mark as well we're both noticing some of these these bugs so they've got to work on version 3 but if you're not doing it if you have for example the Windows Firewall disabled in the root oh you get rid of that get you guys to move it to a

sub 0 you so that way if you need to turn the windows firewall on you can drag the system out of that oh you drop them into a no use it's got all this stuff turned on and now the sudden you're logging to get really enhanced and it's nothing more than moving the machine to a different group and put it in any you won't block anything the user will never know and now you can start collecting all this information a highly valuable new service in the case of winnt I 2012 the change service we were CPL support what's a change so it's a 70 40 and if you're doing PowerShell logging you do need to have a default

profile yes there's a bypass for that I understand that you just execution policy bypass and no profile you won't get this to work but if as your people use PowerShell normally and you set the remote signed execution policy when scripts are executed you will catch what the command lines are if you turn if you enable this there's a couple variables you have to put in so these are top priority the things to start with yes

there is layers don't do me a lot of good they're very noisy and these bed these apts rarely so he 4625 is the login failure says wouldn't that work it's very little value to me honestly surprising internet-facing yeah higher value but you're going to fill up a lot of failures I'm more interested in that Sam connected to his machine his machine his machine has a machine his machine because we know just when their behavior they popped the creds they know the creds failure is not an option for them it turns out and so just very little now there was some times where commodity Mauer will try to brute force matter of fact some of the malware reports one of

them has a whole big list of the top you know thousand bad passwords that had brooded so yes but be prepared to have a lot of noise that people are actually doing this with failed login so that's a good question it is no windows login cheat sheet but just be aware it's not my high prior highest priority here's an example that big list of commands ie exact should be in here as well it's not on my sample I took a couple out just shrink it to make it readable but here's kind of what my splunk or any log management it's event code 4688 look for all these and then think about it from the perspective of

well I execute these some of these all the time you won't do it in the quantity you see in the tax no way you just one size matters so this was a big one the fact that this payload got dropped in the registry turns out if you scan the registry for fur keys bigger than 20k there are only about 20 of them so if you filter those out in a whitelist in some way and you look for large keys the win NT I was 296 k and so my buddy had written a little script and we ran it on all the boxes and now we found every box that had large keys and then we filtered

those out as the ones as normal like your icons are all that stuff stored that's a big key for example Microsoft stores all kinds of status in there yeah you can find this up pretty well there is a GUI version reg scanner from near soft and then David longenecker in Austin went Austin peeps through some conversation of Twitter went out wrote a python script to actually do this I think the defaults 20k to look for large rich keys we are also incorporating this functionality into log md this is a big one where I want to replace using a GUI so log in D will actually do and provide you a whitelist for large reg keys as

you use moving forward enhance logging system on if you haven't played with it please do turn it all on but warning somebody noticed this yesterday was using it you must exclude sis mom from your AV or will kill your box because now is it's registering all these dll's and drivers being loaded which is boatloads AV can keep up and your system will just come to a screeching halt so there's my warning to using system on with the monitor logging with the network logging and again the windows logging service is system on on steroids yeah his thumbs up over here I can't speak enough but I've had input to this Jason McCord is added PowerShell command

line logging he does it at the trace level not the logs it's actually a replacement to your logging agent and he's doing a trace level not reading the windows logs so you want to do both great and it sends it to it Eric's syslog server he's also with the next version I proud to announce that he's doing wmic commands so if the bad guys are launching wmic you're gonna be able to see the calls that they're making so at least maybe now I can see if they're using that functionality I'll see that but a lot of the hackers have moved to the PowerShell execution of wmi and that's a different problem that we have a solve with with the future

project and working on but yes enhance logging please take a look at these now we're discovery I can't speak enough about how well it worked again simple you have a suspect system so go get a good system hash the hell out of it and run shaw one deep login d will also have this functionality to replace Sean D and I basically allow you to compare a suspect box to a good box get a short list of hashes to look for and help you find the bed the bad stuff before before they get too far then how many people here have a log management solution yeah so well you don't have a log management solution so real quick that innovation

was to co my my colleague up introducing new omd this is a tool that we wrote as an outcome of dealing with this kind of crap i needed a command-line utility because there were cases where I don't have log management I do some dir on the side and I wanted to speed up my lab analysis so with this we wrote a tool and it's called log md it's at log md com there's also a link to my website now archaeology calm it's free please download dumps a report to yeah we believe in and the community needs some of this basic health we're Richard developing up with some other stuff that may be a small cost of the item because

of a development required but it is based on enforcing you to run and use the windows login cheat sheet when you run this the first time in a box that's not properly configured what I say is properly configured for that or the windows login cheat sheet it will fail and it will not run until you properly configure the box I'm not going to collect something that's going to be garbage and windows by default clicks nothing that's of any value so once it's probably configured boom you get a spreadsheet when it does fail you get a nice audit report of all the settings with their currently set to against your machine comparing to the windows login

cheat sheet and also comparing to the Center for Internet Security and tells you all the reg keys to tweak and the profile for a powershell and the very you have to set the idea here is someone says I have an ironing I our report going on right now can you help us out well you have logging you have this you have this I don't know no I can send you this executable get your guys to get it running send me the results back and then I can bid on you and depending on one you take will depend on how much I charge you if you should be able to do this in an hour if you take two or three

days just to get these settings turned on yeah my price for you just went up which means it can be really hard to work with your people so it's really good for that kind of thing that's how I use it it's a great customer cloth it will help you assist in tweaking file and registry auditing as well so as you turn this on one of the things we collect is what you're capturing in the logs for file audit registering so example I go and turn on auditing in the services key and I let it do all subkeys the tcp/ip key is very noisy a couple others are also noisy that's too much data it's not ever going to help me so

I'll go turn off auditing in that location so now the amount of data and putting into my logs just shrink and the stuff that's in there is more actionable for me and it will help you do that too also with file auditing oh I turn this directory on way too much data I'm going to go turn it off here here and here that looks good because you can't audit everywhere file and registry just can't you have to be selective how selective using our management to guide you you look at a different way for that one but maybe no you really couldn't but yeah again large ridge key that I do everywhere so yeah we have multiple ways

of looking at this problem and then once the system is configured you clear the logs you infect the system you run login D and then you view the report so expel Excel spreadsheet yes I'd understand use the filtering it's awesome but again we just started this project in May and it is free and release now that's pretty fast and it works pretty well as I showed some people yesterday anybody in here play with it yesterday would you at least look at it and saw what I did with it shout out what you thought and then again the audit report for you auditors or somebody says well I don't know what I'm set to but I just want to see run it

get the report out of it add it to your assessment report or if you're a consultant use it to say hey I want you to of your logging and here's how I did contribute the Center for Internet Security benchmarks years ago I was H at HP so that's why I know some of the shortcomings of that 33 whitelist so I run this thing I got all this broadcast traffic I got all this normal printer traffic for example great we give you a whitelist to get rid of the IPS that you don't want source destination destination port so you put that in there in any combination and it will filter the results it does not just drop

them it puts them in a white listed report so you're not losing any data you're just moving it out of the main report into a secondary report we also do it by process command line this command line convos blah blah blah I never want to see that it's never a value whitelisted this particular cam man running the search indexer never want to see it white listed so you start chopping away at traffic by using the whitelist in a lab my run of log md in my my lab box is almost blank because I white listed all the normal stuff that happens in my lab so when I affect the box i'm pretty much seeing all the

interactions the box is doing within that malware and then again process name don't do that overly random make sure that if you're executing if you're excluding my process name make sure you'll never see anything in the command line that's a value to you because if there is at some point in the future like you would want to get rid of sysprep because all those disparate command lines would be of high value so be careful when you do it by that do it by command line first only if processed name by like I exclude process name log md only do it when you know there's never going to be value file auditing whitelisting as well so that way as I'm

looking for those reg keys and I'm trying to get rid of some of that noise i'm turning on auditing and kind of doing it quickly in an ir scenario i can white let's those out of my results and then it that that white list becomes what I use to refine splunk and blacklist the stuff I don't want to go to splunk that's noisy but i do want to collect stuff like that and then the report itself so here's a typical crypto event someone brought a laptop in and said hey I'm cryptid so here it is in yellow not something I Iran it's a box that was given to me there's the execution of the cryptolocker stuff it

immediately copied to a different file name it went through the deletions spit that data out to null same file name here same folly same here so it's saying delete me but don't tell anything about it and then of course after the cryptolocker it executes savvy SS admin delete all volume shadow copies so you now you now know restore is the rebuild is the only option here there is no volume shadow copy option to restore from and so that fast I know what to do the box boom done and that's pretty much you can see that the lines 40 through 72 there's not a whole lot of data in this so it works pretty well for that kind of

thing to investigate so here's a word doc I exploded into lab there's the word doc being executed here's the batch file that dropped out of the word doc this is the thing that the guy had yesterday his was 10 dot exe and this one's 9 eggsy it's pinging china because it wants to make sure it can communicate or it will kill so i've tested that and it kills if it does not complete this and then there's the nine dot exe executing all over the place there's the ping once it's a done only once to see the episode still communicate still can't communicate and then BOOM I move over in the spreadsheet and I now can associate

through the windows firewall login 5156 I can see the cool thing about windows firewall logs it tells you the process and the IP port and protocol it uses so here's windward talking to an IP this particular IP is a dream host server I'm sorry but word much like PowerShell should never talk to any IP ever externally windward should never talk to anything but Microsoft so immediately that got added to our iPad IP list and then see script Ram we have some more IPS and then there's the ping running and then there's the nine and sysprep running and so I'm harvesting these IPS checking in with who is so that I can see what company and network they belong

to I add them to my black litter into my bad IP list and splunk I run it anybody who visited those IPS is now infected I can immediately remediation that information and this took me 15 minutes I did that whole whopping 15 minutes so it's it speeds up that kind of process for me again it would take a little longer for the advanced stuff because i have to pick it up from here and collect it from there and put it here reassemble reboot and all that but this helps a lot this is really shocking how well this worked for me so that's why we're sharing it it's kind of one of those hey this thing actually works but

yeah really really well so in summary Mauer is noisy we can detect it logs hold all kinds of information if you just go out and enable and configure them and the windows logging cheat sheet there it is what it looks like six pages of gold talks about how to enable configure gather and harvest and able and configure self-explanatory gathering is what you can do it like command line with powershell or w event util command line utility and harvesting is talking about how to put it in a log management and regardless sections and so please go do that as well as get the windows lung clogging cheat sheet to help you out your smudge get you started if you're

not doing those kind of security alerts and with that here's some resources Maori archaeology calm also has a tab to log in d log dash md calm will be a new website next year since this thing works well we're actually going to split it off and here's we get the stuff pretty simple you google this pretty much Google will help you autofill archaeology because you can't spell it and you'll be able to find me real easy actually also on SlideShare you can find me through that as well with malware archaeology calm and let that I'll take any questions and Sam will throw Ding Dongs questions got one right here we did within our environment yeah we could

they set up a box and it was talking to that box it was how they communicated with it and so we used a minute yeah we were able to get in saying haha boom we focused on that box ding dong yes so before log md how was I pulling this host large in the environment in gaming I had splunk unfortunately sometimes we had to go to the box remote or I would execute bigfix scripts just basically scripts that would collect information specifically I was looking for with some of the tools that I have they talked about my malware discovery class before that very manual so I've got like a script that I run around reg scanner and

a GUI pops up those look for big key so we ran that across the environment redshaw to capture the shot and try to compare that to others a very manual process of just executing a lot of sis internal tools so log in DS I'm trying to write that to replace some of these more clunky mechanisms so in theory you could use s sem or or whatever agent or a net used PS exec or however you want to execute this executable throw it on a bunch of boxes are suspect run it take the output collect it and then you get rid of a lot of steps that we're doing very manually this is kind of a

manual ARR people do it Mannion people would use mirror and a lot of these tools that I'm using but a lot of our guys kind of do it this way is very unfortunately manual unless you have something you know big and fancy yes any other questions and then going

do I run into Mauer to try to disable logging good question surprisingly the only dumb things the Chinese did in 2012 was somewhat clear the logs it's clear they had a noob running along with a expert a lot of mistakes on these systems no mistakes on these systems if you're using log management the stuff gets sent to a log server they can they can muck up the local logs all you want if they turn something off we get alerts for that so if you start messing with turning things off we get alerts so that we have to look at the condition being normal have they are do they are have some things hung or whether it was them

or not happen normally absolutely and but we have a Splunk running correctly is big fix running we use each product to look at each other product and so we have this cross thing saying if splunk running in big fix is big fix running and splunk and we kind of keep that stuff up and up to date so yes I really not because I don't think they want to trigger the there Carlos always says I got you to click on the email and now I got to gather as much information as I can figure out how you're doing it and by the way his recommendation in his present during Khan was referencing the windows logging cheat sheet he said if

you want to catch me you need to do stuff like this if they go after that if Carlos let's say in a pen test starts going after that yeah you got to look for the conditions of clearing logs of stopping a services and I do automatically monitor that you saw the 70 45 and 70 40 in there of new and stuff so good question I wish it was that easy they don't seem to do that and normally when you see the behavior of them getting to that box so if you can't you just rootkit the box sure you can but how do I get to the box without making any noise in any way putting something on the bars getting it

executed without doing any of this stuff that you saw me collecting in the logs it's it's really difficult you windows does a pretty good job of collecting stuff I mean really there weren't any gapping holes and all that apt we saw I've just never run into it where Wow we're missing a lot of data tripwire missing tons of data but logging built in logging not so much i love tripwire by the way it's just you have to understand its use really like any security tool yes

ah so after they infect the box it phone homes it turns out these guys these actors that's really good question I didn't mention that in the talk when they infect a box it doesn't necessarily phone home they are heavy users of timer the timer API to where they will infect an environment and wait until we're gone in the day and sometimes these things wake up at seven o'clock at night so once we understand the IPS we can watch their behavior and they're very much opposite us either you know the Russians or the Ukrainians or the Chinese over the pawns either direction they're opposite us and they really do most their work at night because they know

we're offline and so it's a better chance of them getting further and embedding themselves faster without us watching them i would say eighty percent of the work happened off hours and did not phone home for at least hours days or in some case 13 days they were really good at just not making a lot of network noise for that reason of us looking at network traffic

yeah the the the logging that goes on in the whole gaming platform environments are very detailed they collect and kajillion pieces of information they can tell exactly when one who wear all that stuff and they will restore accounts very common yeah it's probably because it came from his IP probably they know that and if your IP is compromised it has to come from another IP from another country for them the gaming is really good at that yeah yep any other questions I'll be around talk to me and got the big picture