WarGames

okay this all introduced Justin while we're sooo late it's too late in the day thanks kidding are you gonna give away prizes too or are you boy prizes oh okay so he gets away give away the prizes seriously you should always see him for after the talk

[Laughter]

she been asking you for the trivia questions on wargames okay all right so really quick Justin over the past 20 years he's worked in a variety of places supporting educational nonprofit organizations all the way to government super secret squirrel and fortune and fortunately 100 companies this is a part that I like the best in his spare time Justin enjoys pina coladas getting caught in the rain and he's a Sagittarius please welcome Justin coke

you're all still here and maybe only partially buzz I'll try to be a little quick on this originally when I pinged and want to say thanks but what are we going to talk about because we were supposed to get this speaker section right for being a sponsor and they're like well you need to have a real talk and I was like oh like I can't just tell you like what our company does I'll have to come up with some things so we took some more stories that we think are kind of interesting and some outcomes I'm a big fan of wargames and you know always like Joshua's kind of insights right on you know fundamentally what's going on

so that's where everything's tied into I'm one of the owners strong and Chad back there one of the owners of Thea I won't say too much about it we're professional services consulting company right so we help companies and organizations really secure themselves for lack of a better description if you have more questions and come find us we're not a staffing agency a few people were saying that up furnace went back there but we don't make products we don't sell products that's not really our gig so first off I want to thank everyone that's involved with besides our to you I know it's not easy to put together an event like this I've done some volunteering and assistance and

speaking for the Portland PD X over in Oregon besides where I live and I know it takes a lot of hard work a lot of effort to bring the community together and really fiha our company just started having folks here about we've about 10 strong and we're growing and so we really wanna be part of the community too so thank you guys thank you I know Greg Patrick and Pavan were folks that I got to interact with but I know there's a lot more of you out there so thanks again and keep supporting the community there's gonna be maybe three four war stories depending how much time people want to give me and you know I'll be

available for questions or we can have a beer afterwards and chat - there's a fair amount of anonymity for the organizations involved so I'll give kind of the lay of things that happened and what some of the takeaways that we have from it so the first war story I'll call uber espionage at the time it was a fairly novel approach that we had seen where a particular set of folks were targeted at the hotel that they were visiting at so you know a lot of times you put in right like over the Marriot there what's your last name what's your room right and only certain individuals of that set of folks that were traveling actually received kind of an extra

special package so not only was the Wi-Fi network of the hotel compromised right by the third party that was providing it the people there were being directly targeted depend on who you were and as soon as that person put in their information right and they needed to receive the Internet they were told they needed to update their system right for Adobe the interesting thing is that it really was an Adobe update and it was signed it wasn't Adobe certificate but as far as Microsoft was concerned it trusted it was a trusted certificate that had been stolen to sign and it had some special editions that right like a backdoor a rat key logger so it was

pretty interesting that also as soon as it had access it began immediately establishing another foothold persistence numerating the system and also what was around it and collecting orally searching and then collecting anything that hit a specified file list in terms of keywords so what happened we detected and contained it within about 24 hours on this particular team the the reason we were really able to do that is typically when certain folks were traveling we were a little bit more of a high alert we had a pretty good understanding of certain things that should we not be on our network especially around certain what we call high-value data sources and high-value data within those sources so we pay a

lot of attention you know I know everyone once you know zero trust software-defined networks was not really a reality so the first place we go when we start working on things is at the network like DNS you know web traffic email traffic and looking at that when we see abnormal user agent strings a lot of times that's not matching what we normally see for what most folks use it's either unauthorized software or it's something bad like a piece of malware trying to look a certain way so we fire an alarm off of that have like a human being go look at that a bit more right and in this case it was a legit problem right this this malware calling

back out and establishing its tunnel and then encrypting it was was not legit right and that's actually what kind of tipped us off in the first places there might be a problem so from that we we made phone calls because at the time actually we didn't have pre-positioned tools on the systems to do like an incident response kind of you know if you think of like me or fire H X or carbon black or in case or whatever thing you might use or leverage knowing we had at the time was a it's now IBM Tivoli endpoint but big fix so kinda like a precursor to tain iam and things of that nature so we get some limit information with

folks though they were able to reach them and have kind of walk them through how to do you know the guerrilla tactical incinerations collect this do this do that so we found a lot of the the badness files right that we're talking out so that we could start to do analysis and this was a fun one because within like 48 hours of constant right like we were going nonstop we're able to figure out and decrypt with the decoder because we had the samples what that traffic looked like because for us for a while on the pack after was just you know encrypted traffic didn't mean a lot but after we were able to decrypt we

could see what commands and sessions and files were going back and forth and what got enumerated and taken off the system also using the intelligence gathered from the malware samples the certificates used to sign the call back domains the Whois data and doing pivot analysis were able to pretty quickly build out a graph and associate the data for Mac virus tool and other places pretty likely affiliated right infrastructure related to like kind of bad comm that was being used other instances where malware might have been uploaded right that matches pretty close to the signature and other TTP es or IO C's or whatever you want to call them right in terms of very close mapping or

just really Whois data to showing up the same who is registrar's like that's really interesting you must be probably part of bad comm and then bad one bad to bad three also starts to show up another malware that's submitted right to different places but we had initial triage in attributional 72 hours it's one of the few times so I like to highlight it that that's actually the case usually most places it's days weeks months years watch a couple of the other worst stories that that was the case but we were fortunate because we were able to be pre deployed and that gets into some of the Joshua insights I've heard a few of the speaker's today miss Maureen who's kind

of absent but in the afternoon talk about kind of being pre-positioned and we certainly were and we always focus on picking something in this case is Incident Response and reach detection and contain we're kind of the biggest concerns so we focus deep on that data and tried to position as many tools at the time as we could in particular the network we had to build kind of a platform right on open source with burrow and snort but then we used spelunk was something we actually buy and pull together and then we leveraged everything else we could get our hands on right that the IT environment had which was things like a proxy authenticating we had lid pcap

just running to do our own packet captures in certain durations in time right and eventually got a real product called Solera at the time that Symantec owns now but then we had McAfee and other endpoint security things that we'd leverage - we did have a good team people that helps a lot and we also had some preset flags and things we knew we wanted to look at right when they came across our path to pay extra attention - and that's where I think that kind of high-value target data rate comes into play where do you focus in at if you have host data or you just have data out in the cloud or IOT certain data aspects

if you have Network data you know what what's the most valuable and for us we really focused in like I said on Web DNS and also really actually mail traffic to you from SMTP so timeliness having the ability to query and get that data it was really critical - so whatever you are focused in on and if you're gonna use a dashboard I'll put in quotes right like make it meaningful or useful for the person looking at it just like storing it you know storytelling you need the audience to have what they need and then I'll stress that a thing that helped us there that you'll see a theme throughout this and I completely agree

with this zi scalar guy on the software-defined network kind of stuff right he's kind of saying you have like you need to have identity management segmentation and also like layer 7 app segmentation network segmentation is dead it doesn't stop anything as soon as someone gets in they find those credentials they move around just like anyone else and so unless you start thinking about it at that layer 7 like approach whether or not you're going to keep certain network controls and things you have to have identity management segmentation and in this case we had actually to control lateral movement something we've got an IT guys to do was randomize local admin passwords so we see a lot of places still sometimes

it'll image all the devices right with the same local admin password so as soon as you get compromised you move a lot really really fast that same admin password especially in Windows environments and that's a bad thing and then inevitably you find somewhere where was on that system or touched it or something using the domain admin credentials that's how things work right for backup or maybe you're doing some vulnerability scans here to people right using domain admin to go check everything it lays around on Windows environments like you know just can be waiting for the taking right so and then once you have that it's pretty much over but in this case we had pretty good

segmentation that's helped contain right so one still got in and someone's always going to get in but it helps contain everything or story - this one's not over cool but it was really persistent and you can see up there the first thing it was a multi-year I'll call apt state-sponsored whatever you want to use as a term sophisticated group that was well funded a lot of indications that you know that it's state-sponsored and it was a long response and recovery effort - so the first phase I'll call it right we don't know how long it was months probably maybe even a year because by the time we found out about it we didn't know we're patient zero was

we were tipped off about it and so we started looking around you know this is kind of post-mortem these details right and we saw later actually from monitoring that particular group and other intelligence or information were able to gather based on what we found you know that they used fishing in water hole attacks quite a bit right and they even had during the time we started monitoring he wound up using the same technique he's got on other systems later on during the overall local life cycle of this investigation phase two you know really I'll say is once we were able to observe and we knew we had a problem we had had forensics devices now

deployed right on the network that we could kind of look at traffic and starting to collect other things we knew at this point where their access points into our network war work for many places but we were pre-positioned to watch them and so after a cycle every three months I'll calm the b-team would come in and do systems administration they'd make sure the passwords that they'd stolen before still worked they'd figure out what the new ones were they check all of the implants to make sure they're working they'd remote into the other hosts that were within the infrastructure and we wash them it's actually how we found a lot of other infected hosts ultimately because while

they were doing their checkups on there like list of I've got these systems and I've got these accounts and I've got this going on we found more stuff right more badness and that's that's what surely while we were watching we wanted to know what all they knew and what they were doing but also how are we gonna fix this problem right this is this was a large organization tons and tons of systems and users so it's like if you just kick them out are you gonna keep them from getting back in right it's easy to kick someone out or when you think you know like I'm gonna you know reimage this machine or I'm gonna go to

this malware or disable this account or reset the password but if you haven't plugged the holes or at least on something they're gonna get back in so that was really fun it was fun to watch them you know use our own tools sometimes install updates even for the servers and things so that people couldn't get in and then Phase three actually I'll say is that we observed the a-team I'll call them come in they used very these the same systems to come in to the network but then they deployed in pre-positioned new tools for enumeration key logging password stealing and they had dedicated scripts and processes run through things like PowerShell SMB write or they tried to

blend in they'd go through already Pete other systems and they were specifically looking for intellectual property rights searching for research and development design specifications source code a lot of interesting things and in fact they had to pivot from a Windows environment into a UNIX software distribution environment and go after source code repositories so they won't have haven't used some different tool sets than what had been positioned on the windows servers that were kind of their access point because we were watching them and because time and past we were really able to not tip them off what was happy I know one of the earlier cloud discussions was right you don't always want to do the eradication and I agree

with that immediately we were able to keep them from getting the crown jewel so to speak while still continuing this observation and collecting data on them because we weren't ready yet to do our rebuild and recover here right we weren't ready to plug the holes we knew it so we didn't want to tip them off that we knew what was going on and we're watching we were prepared to pull the trigger though so what happened here how did all this come to be right so that's kind of the whole story but initial was actually notification from the government Thank You Uncle Sam it was not a lot of information but enough to get started those were the five seas and

like I said we had a custom Network forensics platform in this case two that we deployed and it allowed us to find a few more things right and then we kept pivoting from there after we had took forever to get like images collected right was from organization organization that was affiliated things like that it can be a really lengthy process to get those disk images all the proxy logs and then there's incomplete records right when it's been going on this long so you just start to get hints and then everyone Somali strike gold when you're in a large enterprise like this trying to hunt things down and the more we got positioned to watch things the more we

would find right so when they came back in they actually helped us the most on that first cycle that we saw after three months of going around and checking all the systems luckily this expedited our deployment so we could deliver more and more visibility it also we were able to convince you know the people in charge that we should have the patience to watch to gather this intelligence and to prepare right so really the preparation was like Identity Management hygiene and hardening and that layer 7 type of compartmentalization and protecting the crown jewels the type of accounts that they had access to were domain admin Enterprise admin accounts right and in Active Directory you own the world then

so how do you prevent that from from happening being spread out so we had to have a pretty complex plan and I think you all know Active Directory or any idea solution is like the basis of everything in your environment right so if you mess with that you can break a lot of things so we prepared and launched like this massive design to kick them out but also to keep them out that's really kind of where these then kind of insights come from we did in this case you know for the security team we'd picked something first to focus on and really push which was having visibility and data and at the time again that was a network forensics

platform but that's how it let us do more once we had that tipper right that we could go look at things and we had pre-positioned at least the network tools we had good analysts there too we also got lucky right someone told us something was going on potentially so we wouldn't took a look and were able to find more and because there was something we were able to do more again we were prioritizing what we thought were our high value kind of data sources since we were kind of that moat design having pcap layer seven and the ability to kind of deploy custom signatures where's Yara or related to burrow or snore it really helped us out

a lot and the data that was useful obviously the pcap who has helped us a lot actually in this case so we could map out additional infrastructure then go pivot back and look and see if any of those callback domains where other malware samples were being used in a couple cases that was true so we found more things and then good I'll call threat intelligence but information sharing that analyst the analysts network can't be underestimated and even official channels proved useful here although slower and patience I was this is probably the the most patients I've seen of any organization to have the wisdom to listen and then watch and gain and kind of progress over this and

take a little bit of risk but also control that risk at the same time for gain it's hard the first thing people want to do is erase remove eradicate write and be done reset the passwords and I am you know you've got to have at any management right if you don't have that figure it out it's gonna be terrible so concepts like Microsoft's red forest if you you know go take a look at that or I can talk to you about it later it's a way to do in and everyone winds up using Active Directory but it winds up working with things like octa you know as your ad same concepts of how you create true boundaries with an identity

management so the same idea of like network segmentation right how do you do that segmentation with your identity management you know and who the people are who the devices are again what the disease scaler dude was talking about is a lot of sense right you really got to have that whether you keeping your traditional network and all those things or you go to his zero to find software network that's key place that has to be segmented and controlled and you got to have really good hygiene so privileged account management that kind of thing on there really matters you can't have these super duper accounts that allow you to have problems just sitting around or you're gonna have a lot of pain third

worst story the apocalypse I call it it was the same kind of concept that you know you have phishing emails they get initial access again it was those local admin rights and then domain admin rights that eventually led to the downfall and full coverage of this entire enterprise network you're able to spread ladder really really fast and you become invisible right I mean it's just RTP or SMB or any other number of allowed communication protocols and tools that everyone else is using and you look like Joe or Bill or Susie or whoever and so silently for weeks even months exfilled a large amount of data like everything I'll say so emails files everything was taken and then they set

up their next phase because I really wasn't their goal their goal was a big more of a fu I'll call it because it was we're gonna destroy everything we can right and you see more of these attacks happening all the time more data you know through either encryption or manipulation of master boot records or certain key boot files right are just there to make things inoperable to damage or lose data for for organizations and individuals and so now the labs that was the last stage after collection they blasted everything with those kind of credentials - it's pretty easy to do so wiping tools and manipulation tools everything endpoints servers obliterated very very not nice what happened well it became obvious Oh

people were like getting another computer systems and I'm like this isn't working this isn't okay there were things down mass panic sets in in Mass there were other indications right in hindsight when you do the investigation because you can although it's painful spend a lot of time recovering these disks if they weren't fully encrypted or files actually securely deleted right I mean it everyone knows you're probably right you can the data is still there effectively it just takes time to acquire it and reconstruct it depending how much damage was done to the filesystem or the file itself so we were able to recover a lot of data over time like months and months and months to try

to understand but that took a lot of hindsight time so there was an army of IT and security folks to create a new temporary network and columns capabilities as well as to investigate and rebuild to a new operational network right so how do we get back to some some new state that works it's really really really hard to recover when you don't have domain controllers they're just gone or your backup systems are just gone right and I don't mean like hey the tapes are there or this other storage mechanism but when you don't have the encryption keys for that cuz they were sitting one server that wasn't backed up the backup to the backup right all of a

sudden like encrypted like storage sitting around becomes pretty useless or even better assume you have that and ability to decrypt it but then you don't have the indexes when you're talking about manual tape and running through and rebuilding indexes that takes a long time when you're trying to get things restored it doesn't work so good you mean you're literally sitting there thinking about things spinning again right at like this speed I can count my watch versus solid-state drives that's really painful so there's some you know thoughts they're right about insights from Joshua at the end and then it makes investigation and it responds really slow everyone wants to know why what are they gonna get back what's happening but

like I said you don't you literally you've leveled right someone set off a nuke and destroyed this place it's really hard to get data to answer those questions and it's competing with the time and resources that you want to get columns and other systems back up to these organizations have to keep working so insights this one was not having forensic stuff pre-positioned kind of caused some problems someone should have noticed at some point when terabytes of data were being exfilled out over their connection that usually looks kind of bad so that was that that hurt a lot right and then I guess also I think it inside is that destructive attacks are really real now not I mean you hear some

in the news but a lot don't get advertised right and where people are pretty pretty harsh toward towards folks now if they have a reason to be and it's a good idea to exercise what a complete you know destructive attack looks like from a coop disaster recovery scenario like because if you lose those domain controllers right or those backup systems or whatever it is that can be pretty like impactful in terms of how things are working definitely I have a backup to the backup and in particular like you know we talked about high-value data or things to focus in it's the same thing with this on the IT side like those indexes are really important the

encryption keys are really important certain things like that right that are that are just the crown jewels they're like the core of the foundation right the cornerstone and DC is having offline copies and frequently having those updated you know and some kind of a backup that's easily accessible or usable is pretty critical typically too and then honestly again I know I'm part being on it but this idea that you're gonna harden in segments at the identity management level and compartmentalization of that matter so much and also focus where you can have the highest impact return and control this neat little graphic up here all the other ones have been movie references but this one has some additional deeper

meaning that's the kill chain there if you don't recognize it you know Lockheed Martin owns that and a lot of people talk about it and then the sub breakdown though is actually more of a mitre thing the TAC methodology there's actually like a bunch of information get that off of might org site it's really cool stuff comes from a lot of guys working on red team's or penetration test and they're doing offensive computing building out enumeration and trying to specify what happens in attacks when you move lateral and those are the areas we focus in and as a defender there's actually they just had a mitre attack con conference that they streamed that me able to pull down

to it's a good place to go look that when you talk about in those areas in focus it's a lot more realistic of where you have influence unless you're super secret squirrel agency you know or maybe like a company with like you don't give a rat's about a you know attack back particular you're not going to do much in the recon phase or weaponize or exploit phase where you're gonna deliver an impact and even if like let's say you are investing therefore threat intelligence or whatever you want to call our information and know what's going on that can be helpful but someone's gonna get in or do something at some point and so once they do get in where do you have

control where can you invest to minimize what happens I used to talk to like an admiral in the Navy about it like well you know ships used to be made out of wood you put a hole in it the whole thing sink so now you guys don't build your ships out of wood right you compartmentalize you have segmentation in it so that way if something breaches you seal it off so that area floods but then you can pump recover save lives whatever right on the rest of the boat they've like you know design these things to be built that way so that they don't just like lose everything at once right now usually when you get

compromised you lose everything at once with domain admin right or Enterprise admin or local admin being the same everywhere whatever that is but again attack methodologies a really cool way to look at things and think through and they're constantly refining it and that's where your all of these tools or if you go to just e scalar and have no tools I guess right as a guy was talking about you would have the most impact to control something happening I think I've got a little bit more time because I don't know how late we ran over but the last or it was kind of tool dump fun so I don't have you ever been somewhere and a

vulnerability comes out in the public and then like people who are really important come run any like what's going on like are we do we have a problem or you know are we gonna be hurt you know are we vulnerable or are we under attack you know or or they here maybe about like exploits or tools being dumped from super secret squirrel places or other types of hackers right and it's in the news it's a really hot thing that happens to us all the time in our teams and people so some important people one of the no you know what what was going on do we have that you know do we see it and in this particular situation the

dataset is huge like exceptionally large we're talking all of the Internet traffic in and out for thousands of large organizations and so this is you know kind of defined if you think of like net flow you know back and forth billions of flows a day there but no visibility into endpoints are beyond that so you just have all this internet data going back and forth between organizations to look for badness and say something it's pretty hard to do it's that needle and needles think someone was making fun of it it really gets difficult right it's a big data problem so how do you build an analytic Fernet flow or we have like snort like seems your capability to apply there to

define in this case you know it's just kind of one on one signature building sill but it's a good process to follow build that profile pull in the lab this thing's been dumped you can have a copy of it build kind of your test harness right to run it automated execute it collect lots of samples and data and enumerations as much as you can you got to really stimulate the environment not to be made fun of but yeah you know you got to get things working and then once you have that interaction and data you know you can start to do something with it right and in this case we were having a hard time because most of it was

encrypted so it has high entropy and that looks really hard to detect more for devices to figure out things on it so our analysts in this case actually kind of started working with some different visualization methods to profile the data which wound up being really successful and he actually kind of extended and built a tool to help a large packet you know and in other types of data analysis to visualize and look kind of the cult of water flow method and I'm talking a little bit past my scope because I wasn't the analyst but that's what lead seeing basically a commonality right outside of all the entropy that was like ah that's interesting let me zoom in on

that and there's a repeatable pattern that we can detect and do something with so this is kind of like again hard to tell probably up there but that banding that's circled was an indication of repeating right commonality whereas the rest was pretty much randomness and so that repeating pattern that we visualized we could go then dig in deeper and see what that really meant looked like and it really turned out to be part of the handshake of the protocol cuz it was using a custom protocol for the tool on those being profiled with it and we could then build a seizure around that now it took more tuning and fine-tuning especially with an old massive data set because all of a sudden

you find with a massive data sets other things that you know look kind of similar to this weird protocol that was being used it was a custom written protocol but still I guess millions of people in the world right it was there's several false positives so it got tweaked a bit more but it worked and also kind of really showed the visualization of like really weird data that looks like nothing to humans you know or you can't find normal things because it's custom there's still stuff out there right when you start to visualize and do clustering and things look for groupings it stands out like a sore thumb and then you can go investigate further so that was pretty

cool the ability to you know have someone that can do coding and help extend tools is really helpful I would say I was probably astute but you'll find a lot of places that there's no one enabled to do that right there isn't an environment to do it they may not have the skills or they have the skills but not the environment and the leeway to do it it's really important to be able to extend tools glue them together do custom analysis things of the nature and then enabling that environment is really critical too and I think that is another thing to credit to this that that enablement was there and the ability to work on that and then apply it in the

production environment to write and test it further and execute on and refine so that's kind of the end of my war stories any questions I can't see well with the spotlight so if I if I'm not seeing you might have to like wave at me like this or something I'm gonna go with no that's a no there's no questions everyone go drink eat dinner thank you for coming

[Music] thank you again it's not it's not easy to wrap up the show who last talked so I appreciate you talking and I appreciate everyone hanging out here we are gonna set up now for hacker jeopardy so I'm gonna actually before we do that I'm gonna plug in while we're setting up did everyone just leave me come on John John get some you know come down here help me Oh everyone oh no everyone's play no everyone's playing