0Day to HeroDay: Bringing A Company From Scorched Earth to a Modern Security Organization

Ryan was new ski Ryan has been fighting evil for more than a decade with expertise in driving security transformations with both small-scale and large-scale organizations Ryan leads various tactical and strategic efforts to ensure the continuing success of his business forever a student Brian enjoys learning from others sharing his knowledge whenever he can and invites folks to stop by and say hi or reach out on Twitter @ ry underscore whiz so with that being said I'll go ahead turn over the mic cool all right we got to get more people in here so on three we're all gonna yell everybody get in here okay one two three everyone get in here I got one oh we got

another one all right so like we said my name is Ryan was new ski yeah I do everything security everything IT if you guys have a hard time there's a couple of HRT's on here feel free to go to the twitter i got a pin tweet there with the slide deck there otherwise this direct link right there go ahead download them check them out on your laptop if you can't see the screen but uh we got a pretty big screen today so today we're gonna talk about ransomware destroyers basically anything that could go wrong with your environment to the point that you're at basically a desolation so a lot of the examples I give they might

sound exaggerated they might sound fictitious they're not these are real-world situations a lot of small businesses a lot of schools a lot of charities they have underfunded IT and non-existent security so we really need to think about what are we going to do in case we do get hit by something that just absolutely devastates our environments so since I started giving this talk there's been a couple news stories anybody not Pat yeah okay couple couple were you impacted or just know about it okay okay so yeah it's pretty bad one about ten billion dollars estimated right now mayors on delays a couple of our FedEx got hit it was a bad one anybody see this one on Twitter so

there's a small email hosting company via female they're very transparent about their instant response but essentially they caught the attack during the middle of the white basically and he's just writing zeroes to all of his backup servers they really haven't recovered they lost all their customers emails anybody working oh seven oh eight right now you get your spreadsheets out by the way protip biggest blue team tool Microsoft Excel a couple news stories this week Florida Baltimore you guys hear about these anybody trading Bitcoin around here anybody watching Bitcoin what's it at today eleven twelve yeah so here's another talk anybody that wants to give a talk go submit it right now ransomware occurrences with the cost of

Bitcoin so bitcoins back on the rise you're gonna see ransomware coming up again so these things actually happen so although that this is definitely a hypothetical never happened I swear let's take a hypothetical so you're sleeping but two o'clock in the morning you get a text message from your boss says hey a lot of people are complaining stuffs down you take a look at the network it's probably a firewall issue you log on a server and you see this maybe you see this one how about that one anybody know what that one is wanna cry anybody know is a little important or maybe you just see this in the case of VF email where they were just writing

zeroes to your hard drives so what do we do we got instant response plans right well we know it's important we just haven't got around to it yet we had bigger problems we had to upgrade our Microsoft Exchange servers first and then we're gonna get to instant response awesome how about disaster recovery oh yeah we have disaster recovery Jim runs it every five years and it's great well where's Jim I he retired four years ago excellent anybody know how this stuff was built do we have any system documentation oh yeah yeah yeah remember Jim all right that's fine we can do this so we are at a point now that we assume that our servers are

ransomware they're wiped maybe it was just a huge corruption on your storage for some reason your company yeah your computer's just aren't working what do we so step one is to breathe this is gonna be awful I'm not gonna lie this is gonna be one of the worst experiences in all your life but afterwards you can come talk of b-sides so they got free beer and uh remember to think okay slow smooth smooth fast that's from the US Navy you don't have time to screw up in these situations anybody know how long their company can stay down before you're declaring bankruptcy anybody all right if you guys don't know use a barometer of two weeks why two weeks I I

can't hear anything scream pay period thank you it's really hard to keep employees if you're not paying them it's really hard to run a business without employees at minimum two weeks you can be down and you're gonna have problems okay so you don't have time we can't risk a reinfection after we start recovering this all right step zero one engage your legal counsel if you don't have one go to outside counsel there are very special things I'm not a lawyer I can't get but you can work under privilege if you engage legal and they engage the incident response procedures let's say you don't do this and the case does go to court everything's discoverable that you found

if you go to legal first you get to work under privilege and not much is discoverable it might save you in the future all right so now we got Incident Response guys because we're underfunded IT we don't have a security team we don't have dedicated IR people so we called out somebody mandiant whatever it is our job is to get this company back up and running so we gotta figure out how are we gonna fix this so we got to say what do we know how do we stop this from getting worse and are we gonna accidentally make it worse that's a bad thing don't don't do that so what do we know this is so I'm gonna

adventure 90% of infrastructure kind of looks like this you got these old legacy Windows boxes that just kind of exist because you haven't figured out how to migrate them yet they're kind of sitting in the back corner they're running some legacy stuff you have some stuff in the cloud because cloud is cool so everybody's going you got clients running Windows maybe some Mac's but haven't seen that a lot of places and then in the corner you got this really dusty black in blue or green it says IBM on there so either a mainframe as400 it's just something that you can't figure out how to move off of economically so it's still there now where we at so this is what we're saying

every single Windows system is impacted by whatever we just got hit by we see some c2 traffic leaving both our client and server networks we see malicious traffic being passing back and forth internally clients so we see accounting come in they power on their machines about two minutes later all of them go black payroll comes in same thing so we're kind of in this mixed situation on if you have a client right now you're impacted but you're not affected so maybe you have the dropper installed on all your laptops and they got a timer on there on next reboot in fact so we're in this little mixed state IBM ERP yeah it's chugging along it's doing okay

cloud cloud yeah we're getting some alerts from there but the server's themselves look okay so I told you it's gonna be interactive what are we doing who wants to go first not everybody wants I'll start off update resumes okay that's gonna be the first one okay well as we want to do give me a hand cuz I can't see anybody anybody anybody there's no wrong answers yes sir Roxy - excellent what happens if let's say the virus is installed talking out see - it's got a kill switch in there you lose connection - see - it just wipes something we gotta think about but maybe we stop them in his tracks anybody else oh we're going bankrupt

quick okay identify okay so maybe this is a commodity hardware off-the-shelf that encryption keys are already up there maybe maybe not anyway else anybody want to do anything on the network all right I'll let you get cheap so we can power it on the system's right if we power down the systems maybe we lose forensics maybe we stop it what if we disconnect the Internet we're bring the business huh maybe we stop see to though maybe stop spreading how did they get in we're not sure exactly yet do they have one of our user accounts at spreading maybe we disabled users sure we power down the IBM stuff it looks good power down the cloud so once we have a

kind of idea of what we can do we need to figure out what impact that's gonna have right is there a kill switch we're gonna probably bring down the business if we're bringing down Internet connectivity network connectivity all right if we disable all users well we got nobody working so we got to weigh these and weigh these pretty quickly because again this is probably an hour one hour to of response we need to figure out what exactly what we're gonna do so most of the cases that I work this is what we do we're disabled routing between the environments we disabled domain the domain accounts not just domain admin but actual domain accounts now that we

disabled their accounts we send people home so something interesting is gonna happen in your office too you're gonna find out that the rumor mill has a mind of its own it's going to get out that you had an incident if you're sending people home and all your systems are dying you are not gonna believe the stories that you're gonna hear oh well the guy that was working on a printer he kind of looked the Russian do you think he was a hacker no you guys got popped 200 days ago and you just didn't know it but um try to control the rumor mill and then go get management to go figure out how to keep this company

afloat without technology management they're good at they're very good at very little actually so they're very good at figuring out how to make money and managing risk it's about it though keep them out of your IT shop so we got this right here we go businesses don't more questions what's broken how bad is it what do we need to fix and what do we do first let's go answer them so what's broken be very careful on how you ask this and who you ask this to because this is what you're gonna get okay I actually got video of one of the breaches that I was on it looked like that okay it's bad so rephrase that went a little bit what

do we need to do to stay in business alright this is actually a better question people can run with this and go figure it out okay most of the companies out there this is what your business cycle is gonna look like you gotta be able to take orders make products ship product pay bills and receive money it's that simple that's how businesses work it's a little bit different if you're in technology but it's all about the same figure out what you need to do stay in business okay now we know how we make money what do we need to do that red teamers anybody I guess I should ask red team nobody blue team couple all right I

don't identify with colors because I do absolutely everything in my job oh yeah those are my people so how do we do these things so you are now on a black box pen test you are going to be scanning the bejesus out of your network you're going to be looking for Colette's you're going to be looking for anything and there's a few ideas of what you should be looking for anybody work in a data center with a raised floor pop the tiles off I found a dr binder underneath the floor in one of these shops and you are looking for anything and everything and at the time don't dismiss anything as old or outdated it

might be on there you might got a pop-by observer that nobody even knew is still existing draw this up on a whiteboard okay this is what we're looking to get to problem is this is what's gonna look like to you this is what its gonna look like to other people okay so again our Blue team handbook Microsoft Excel but everything in Excel things get done okay so we're gonna create this matrix we're gonna say what's our critical function that we're supporting what systems are supporting it and what state is this system in right so we're just kind of walking through a basic system recovery procedure at this point all right have we scanned it is it affected do we need

to restore it do we need data on there have we tested it so we're gonna move forward on this make sure you're using colors management loves colors they don't understand exactly what that is but they know green is good red is bad I put this up on a big projector like this find a TV we lit out of the war room make sure that people can see it without talking to you because the last thing that you want is them coming to you every 20 minutes for a status update and it takes you 10 minutes to draft a status update and you only get 10 minutes to work out of that half hour then don't do that make sure that your

noses to the grindstone because you got a business to say okay now so how do we restore this business well we got a backup system we're gonna put in this quarantine zone because we can't risk a reinfection we're not exactly sure when we got popped we're not exactly sure what even hit us so at this point we need to restore and see if the restore is infected this quarantine zone is gonna be crucial and then you notice up there I got new network we're not restored into old network and this morning Verizon data breach anybody there couple you go so one of the first cases they talked about was a ransomware where they got a domain admin account

and launched ransomware so they up the right square that's good they still have domain admin over your network great like next week you're gonna do the same thing you're gonna get really good at this if they got a domain admin how are you cleaning up that domain admin how do you alright so I'm gonna make another generalization if you got popped by this they got domain admin they went rampant on your system how many wanna bet that they have Cisco Cisco on their network switches or admin Paulo on their palettes and firewalls right you're not sure exactly what you still control chances are nothing it's it's gonna be bad so we need to build out a new network with new

authentication servers depending how bad it is maybe new switches are they rewriting your firmware you're not sure at this point so we got to gauge quickly exactly how big of a rebuild effort this is but we got one problem up here yeah you were running old Windows as your backup system - he's also gone maybe you're part of a wiper where they hit the backup systems first so what are we doing update resume right yeah come on we got offline tapes don't we who here has some type of offline backup that is saving them from ransomware right now I should see every hand if not I hope you have a good Bitcoin wallet huh because what's going on more and more is

they understand that most people are just recovering from ransomware attacks and it's no big deal so they're targeting the backup systems first if you don't have offline that is protected from an actual right you might be up a creek make sure you have some type of offline I don't care if it's direct ask no I direct disk tape nothing don't care make sure that it's protected okay so we're gonna build a new backup system we're gonna rebuild all of our tapes and then we're gonna start restoring into the quarantine quarantine quarantine system out-of-band air-gapped and that is not in quotes an air-gap that is physically air-gapped it is in a different rack with no communication to

the outside world you have a monitor physically plugged in there's no RTP you cannot risk a reinfection i'll say that over and over again because right now we're probably 24 36 hour into the incident and we still have nothing right we're just starting to restore system by system monitor for iOS iOS sees indicators attack indicators compromise and verify that that date is clean then we go to new network all right I usually I'm gonna start bringing candy do these get you guys talking what do we want to build a new network because there's a funny thing that happens when you get breached guess what happens to the pocketbooks of the executives we can go build whatever we want so what do we

want to build oh you guys are still updating your resumes what do we want what do we want yeah Network segmentation excellent we can't spread to all systems if all systems can't talk to each other oh that's what guy oh this pick on you I heard somebody know yeah you guys on next all the cool tools that we couldn't buy before what's your number one tool you can't go wrong

okay he's driving insecurity on you no excellent what are we gonna use it for monitoring traffic excellent something right we probably don't have anything if because if we had something we probably would have noticed it we're monitoring traffic all right we're gonna rebuild our Active Directory because that thing's gone oh so we got here we go chichi so we got network segmentation new Active Directory let's go get us in go roll in ELQ if you want to roll in a security onion go boom I think spunk has a 500 gig trial license go throw something in make sure that you have something because here's another cool thing that happens anybody here have kids what happens when you take kids

toys away what's the only thing they want back now all right we just took the attacker's toy away the attacker wants back in so they're gonna come back this time we're gonna be ready for them multifactor you probably got popped because you had OWA exposed single-factor password spray for let's hear what are we in summer 2019 they got 30 users and they tried each one oh look VPN is also single factor therein not difficult multi-factor that stuff patch managers chances are we got hit by either pick your favorite Oh 867 17 1007 o8 we didn't patch that's why we got popped patch your stuff check your patches with vulnerability scanner got necess you got Qualis you got free stuff

open vas do something anybody here running IR toolkits EDR with like full shell access to your clients if you're not check out grr it's made by Google Google Rapid Response it's basically a rootkit that you install but you can take remote memory forensics so when you do get hit again you can grab the forensics from your remote sites even really good stuff you can kill processes and stuff yes sir Google rapid-response grr and throw up HTTP proxy email protection you name it right now here's the challenge that we're gonna have we got a lot of work to do now so and we're also running the Recovery's in the background when do we know when to quit

so can you perform your critical functions are you getting paid for the product that you're shipping that your customers are ordering right how much longer can we keep the business down we can put in a lot of stuff because we don't have change control because we don't have to worry about breaking systems they're broken already right so how much longer can we keep the business down prioritize your work based on how much time you have and then how much more can you do without any sleep guarantee you're not sleeping it's gonna be my record is 168 hours I think you start seeing stuff it's much better than beer like if you're a cheap date to stay

awake to find these ahead of time so you don't have to think about this when you're in the ninth hour stupid tips of recovery three to one of conferences everybody know that one three hours of sleep two meals one shower yeah these can become negotiable your war rooms gonna smell sorry war room get a war room manager get somebody to take care of the team basically manage the caught in the caffeine intake manage the food manage the schedules set up the phones get the TV's working go buy USB drives save everything you're not sure what's gonna be discoverable if this does hit the court systems it's part of one about six years later they actually

were subpoenaed for some old email records and they didn't have them because they were gone but because they had an actual evidence and everything they put that to the court case and it was actually dismissed due to whatever again not a lawyer but because they had everything they could prove that they actually didn't have it and they weren't holding it back from the court well as part secured out of Bank communications the last thing you want to do is communicate new credentials over a compromised email system don't do that if you have corporate phones and you think those are breached don't you them go by burners alright so moving on second half of the talk yeah that was

awful I don't ever want to do that again so how do we do that well again I like asking questions so where are we worried weak and how do we prioritize to work so it's this peas in the room anybody got a few hay gap analysis so we're gonna look at where we are well we know where we are we're we're pretty bad we know where we want to be want to be pretty good how do we get there that's what we're gonna talk about so where do we want to be you can try to define a framework on your own or you can just go steal somebody's hey come on we're all hackers here for all pirates just go steal

somebody else's work they do a lot better anyways okay good so I stole mists and I made it really pretty for management so here's one of the eye charts feel free to steal it I've got the link at the bottom but basically I broke down NIST 853 into these very easy to understand control cards and organized them by you know your response phases and then I threw down a little 1 through 5 tracker on it to make sure that we are where we want to be so here's the zoomed up one I saw a couple cameras feel free I really hope you're not recording me alright so here's the zoomed up version on one of them so you can look at

anybody reading this - 853 it's wonderful if you can't sleep yes you know my pain it is awful so there's a whole section on physical asset inventories annoy things you have to do or you can just say physical devices are inventoried and rank it one through five okay so we decided on this one we know we don't have any inventory system so we're a zero we want to be a three though so let's talk about what a three actually means so capability maturity model against ESPYs systems they teach this all in the management the boring stuff so we talk about one through five one is a very ad hoc process this is a single person defining what the answers

gonna be okay so this is really typical in small shops where you are making all the decisions step two is you kind of know how you're gonna make the decision you have a kind of framework but nobody else does so it's still dependent on you number three is an actual defined process you can take a playbook or you can take a process and hand it to somebody and they're gonna get the same results that you did if you both follow the same process number four is managed so this is a process where we have exception reporting and alerting and we're finding out where the process screws up and then to get to five is we

take that feedback and we feed it back in to product development or whatever portfolio we're working on and we continually improve this process okay I had it on the last slide I think but uh not everything should be a five if you have everything as a five you're probably miss managing your budget and your money because every dollar we spend on security is not being spent on making money for the business so like it or not we still are a cost center we can try to change those words around as much as we want to get whatever budget we need but at the end of the day we don't make the company money unless we're a security

company okay so we're at zero we want to get to a three how do we do that well we throw in an asset management program right we set up some pretty lofty goals on what we want to do you know we if we have an asset we want to know where it is at all times simple as that so we throw down some basic objectives and we do this for every control what happens is you end up with this portfolio you'll notice that like five or six cards full in two asset management you'll notice a thirteen might fold into boner ability management and there's more these are dis examples but there's more groups out there and

we're just going to see is you got a lot of work to do but if you get all this done you have a pretty good security organization at this point so again we're coming off a week three probably of recovery what do we do first now this is where we're threat modeling alright this is how we prioritize our work so we're gonna there's a couple different models you got data definitions you got data flows you got any of your relationships we can make it more complicated and may talk about the geopolitical relationships the Middle East right we just got hit with a basic ransomware we don't need to worry about that this is what we need to worry about

okay we have things we have things that are protecting them and then we have bad guys yep so let's break that down it's kind of a risk-based approach so got this idea of loss expectancy so on the X we got likelihood of something occurring on the why we have the impact of the occurrence so anything on the top right it's gonna be a big deal we're gonna lose a lot of money all the time that's bad we need to fix that bottom left we're gonna lose a little bit of money never don't care don't worry about it the other quadrants are a little bit more difficult so let's say we're in st. Louis and we're threat

model in an earthquake yeah if it happens it's gonna be really bad likely how did about happening probably pretty low and we're gonna bigger problems if that hits right but I don't like I name a lot of time on those and figuring out exactly how to justify it so I have at one model that takes us one step further I call bang for buck so we take that loss expectancy that we just calculated by taking impact times likelihood throwing that on the Y and then the ease of implementation on the X so on the Left we have very difficult project on the right we have piece of cakes now what we have is top writers a no-brainer

these are things the easy to do and they're gonna mitigate a ton of risk anybody here running laps local admin password solution how hard is it dead simple right PowerShell script GPO you're done anybody that doesn't know elapsed is Labs is a tool provided by Microsoft for free what it does is it rotates all your client local admin passwords so if you don't have this chances are you're running with company password one two three exclamation or something as your local admin shared across all your clients so if I get on one of your clients I dump hashes I now have the password for your local administrator on every laptop and maybe server laps rotates that automatically

for you it gives you a little GUI that you can look up the password for the client at a given time you can press the rotate button so if you do need it it's there but it gets rotated you're different you're unique it's dead simple it's like a three-step process put it in change control get it done next week that's a no-brainer big projects top left these are things like zero trust micro segmentation these are things that are going to be awesome but they're going to be hard okay bottom left again don't pursue don't care about those bottom right are kind of things like when we have time they really don't mitigate much but we know

they have to do it you know these are like cross-site scripting on an internal only only visible to internal and it doesn't really impact anybody yeah we know we have to do it but when we have time we'll clean it up so cool now we have what we need to do we need we know what we need to do first let's go do it so we're going to obtain resources we need people all right whether these are six-month contractors whether they're full times that's something for you to decide right but we have a lot of work to get done we have to figure out what's the best way to get it done go get the

budget for it go govern the progress so we have an empty checkbook anybody know how long the blank checks last after a breach it's about three to six months then you have to start moving yourself and justifying your budget so go govern the progress right show fancy charts these aren't hard to make they look great for management right so govern the projects reprioritize based on business need continue a gap doubt analysis and congratulations we're all CISOs alright everybody on three ready yeah whether you like it or not this is kind of the work of a small security manager see so kind of thing that you're going to kind of have the responsibility of now because you are the expert in the

company now where you take this from here that's gonna depend highly on business need in your relationship with the Executive Board but essentially we just took the company from absolute ground zero things are literally on fire - you know it's not bad and we have a roadmap forward and we're starting to get stuff done so at this time I'm tired of talking I really like to make this a discussion afterwards you know war stories are fun email Twitter slide shares thanks Sarah I'm willing to stay here as long as you do because there's nobody in here until - cool anybody for the group otherwise I'm just gonna stay up here yes sir

laughs for Windows Land anybody Linux administrators here that has something similar that rotates passwords on Linux I do not write I would custom write it because and my codes gonna be ugly

yeah Labs is all GPO base active directory you know shenanigans

come it was there's a lot of paid projects out there Labs is free anybody else for the group otherwise I'll stay up here so everybody's bored of me all right thanks