Backup ≠ Cyber Recovery

I almost did we'll give it a couple more minutes we'll get started here folks in just one minute it is straight up uh we're one minute to it you know what we're gonna go ahead they're recording this so anybody who's not here might have the pleasure of uh tuning in later my talk track today is backup does not equal cyber recovery I'm going to talk more about that later but that's going to be a recurring theme of my conversation as we walk through the slide deck um oops not that one a little bit of introduction um my name is Rich Iker I'm an engineer at rubric in our specialty engineering organization my focus has been on our security overlay products

that sit on top of our already incredibly secure core platform I've been doing data security quite a while I've been with rubric two years in a couple months but I've been doing data security for a long time about 28 years now um up out of the Seattle Market I've worked for some of the folks you may suspect I've worked for Microsoft T-Mobile Expedia and the likes I've done intrusion detection systems I've worked on IDs IPS vpns vulnerability assessment is where I've spent a bulk of my time in the last 15 years at a very large scale quite frankly um there were times where I was managing up to 6 million assets at one time so

that were scanned on a continual basis literally and we never stopped so I did things at massive scale so I've been in the trenches where a lot of you folks will be working and are working today so I understand the trials and tribulations of what can be thrown at you as you're working to secure and work on these systems I'm very passionate about data security feel free to reach out if you'd like more information or have questions after the talk I'll be around connect with me on LinkedIn happy to connect and continue the conversation offline as well um once again though I'm going to be talking about backup not equaling cyber security recovery cyber recovery and I'm

just going to let that hang for a minute I mean what does that even mean um it's a strange statement but I want to talk a little bit about it traditionally Enterprise data protection Enterprise data backup has been in the realm of I.T operations for the last several decades probably 40 years quite frankly nothing much has changed in that arena in the last 40 years until recently when companies are re-envisioning how Enterprise data protection should and can work one of those companies is rubric we're a Pioneer in that we've kind of turned the market on its ear in the last several years with some of the capabilities we're bringing to Market that really start to differentiate between a

traditional recovery and a cyber recovery from a cyber event recovering a virtual machine is easy recovering a database is easy that's the realm of traditional Enterprise data backup you you'll get a request I need a file I lost a file I lost a server occasionally you'll recover a VM maybe a database that's not cyber recovery that's just traditional backup and Recovery cyber recovery is when you have a cyber event a destructive data event and you need to recover your systems um accordingly and there's a lot of information you need to be able to do that successfully it's not just as simple as recovering a VM we're good now we'll talk more about it as we move

through the conversation um the current threat landscape obviously is not good and it's not getting better um on the left there the top activities group that's from this week by the way these numbers are fresh right off the press so as you can see groups like alpha alpha V black bass to lock bit they're not slowing down in fact quite the opposite they're speeding up um top country United States big surprise we have a lot of activity a lot of infrastructure to attack over here as well and a lot of you know economic advantage to attack I put some useful links on the right hand side there my right um these are it's it can be very challenging to get

good open source intelligence threat intelligence these are all Twitter feeds that if you follow in fact I almost suggest creating a Twitter feed just to follow some threat intelligence feeds because quite frankly when you do it'll start to take over your feed I'd hate to have you find out the first time first indication of trouble in your organization you don't want to find it on Twitter but you want to monitor it to find out what's going on you'll also get very rich and robust threat intelligence these security researchers are very open and honest about their threat and telling their research I highly suggest following them when you can because they are keeping track of new and emerging

threats and trends that you've seen your Enviro that they're seeing in your geolocation so I can't tell you how useful it can be to follow some of those Twitter groups um all right obviously you know these sectors are just getting pummeled look at the health care and the public health 210 incidents um I mean across all verticals are being attacked but as you can see critical manufacturing government I.T shops Health Care they're just kidding brutally attached just constantly and the top threat actors lockbit Alpha V Black Cat And Hive I mean these folks are not slowing down so you really need to get as much threat intelligence as you humanly can against them right now when when there's a ransomware

attack and I work closely with our ransomware response team at rubric I should have mentioned that I guess um rubric has a ransomware response team out of our 5 000 plus customers our customers get attacked as well but the difference is our customers have support for them we have a team dedicated to them I work with this team frequently due to my experience in threat hunting right now what we're seeing across the board is that every one of these attacks is incorporating both encryption event and an exfiltration event so we have to really be cognizant of the fact that they're not just locking the data they're just not denying our access to the data they're taking it to use as

leverage so a double extortion basically they're storing you for the encrypted data they're extorting you with the threat of leaking your data on the public or on the dark web and I'll even take it to a Next Step tertiary extortion they're actually then turning their attention to the folks that they that are in your data your partners your customers your employees they might start attacking them or calling them we've heard reports that folks whose data lost data they're phoning the people they're calling the people whose data they stole to extort them now so where does this end it's not going to end anytime soon we have to take better preparation the state of data security

is bad I hate to say across the board we can all do so much better 98 of it shops out there have indicated that they have faced an attack um so it's I don't need to even really go over some of these statistics you've all heard this it's on CNN It's On The Wall Street Journal it's everywhere it's coming from the board down now so the first time in my career people from sea level to senior leadership are coming to us they want to know what is the status and the reason is these statistics they're losing Revenue they're losing money they're losing reputation and they can't continue to do this it's affecting the stock prices this is affecting the

shareholder value or the personal value of the company and it's really something that has to be addressed so it comes down to the simple fact that businesses are absolutely under attack and we live in an assumed breach world and if these Bad actors get into your environment and they get your data they really got you but if they get in and get your data and your backups they really have you they have you in a bad way in fact this suddenly can be a lights out scenario for some businesses because if you don't have your backup data to recover from it could easily be a lights out scenario it could take you months to rebuild

um we have to worry about if our data is an easy target in a traditional system Enterprise data protection system you might have multiple systems from your disk your sand your disk based backup your backup servers your backup proxies these are oftentimes all connected to your ad infrastructure all running on Windows infrastructure multi-factor authentication is not widely implemented and certainly not widely enforced basically this is becoming a recipe for disaster right now we're seeing this get attacked again and again and again in fact there's ransomware on malware specifically designed to find and attack this type of infrastructure setup so if you are running one of them the due diligence is on your shoulders now to

watch that thing like a hawk if you don't have multi-factor authentication on these get it get it fast because they're coming for you so ultimately the complexity level you add in the next level of complexity which is your Cloud infrastructure and most of our customers at rubric not most many are multi-cloud customers they're in all three major players AWS gcp and Azure so the level of complexity that this Cloud infrastructure is bringing to the table further complicates your security model so if you've already got this type of scenario in your environment you add in your cloud workloads and then all of a sudden the level of complexity is almost out of control you might need multiple

full-time employees just to keep control over this environment and of course this is where you get the uh you know the information you need not always being available how do you know what to recover if you're attacked in these environments if you don't have Rich visibility into your data how do you know what sensitive data could have been involved in the X in the breach and it was exfiltrated and how do you know that you're when you're recovering that you're not just bringing the malware or ransomware back into your environment where it re-detonates in your right back where you started from that's the last thing we want to happen because now as I said the c-suite and the board is

watching and they want to know questions like are we good can we recover how long is it going to take these are questions that can be extremely difficult to answer and they want these answers and they want them soon so ultimately let's talk a little bit about the anatomy of these attacks and then I'll get into some of the common failing points in recovery but before I do I do want to talk about this anatomy of a ransomware attack let me just have one set um as I say rubric has a ransomware response team we have helped hundreds of clients to this point recover from ransomware I have personally been involved in 18 in the last three months

and that's just the ones I've been involved in so we've learned a ton of really useful information when working with our customers as they recover from these cyber events so I want to just cover the anatomy of these attacks as we see it first and foremost they get in it just takes one mistake regardless of the amount of money you've spent on protective methodology it just takes one mistake it could be uh Executive Admin who clicks the wrong email and your 100 million dollar xdr investment just went out the window the first thing they do is they stand up command and control infrastructure they do this to bring data in and to take data out the data they're bringing in is

the Hostile malware the data they're bringing out is your data they're specifically looking for any data to exfiltrate to bullet point number two they really love financial information they especially love your cyber insurance policy if they can find that they know exactly what they can expect to extort out of you um of course during this time they're also trying to increase their footprint both vertically and horizontally they're trying to spread out and they're trying to move their access up all during this time I mean a lot of activity in the first bit and also they're not sticking around as long as they used to either dwell times have come down from on average of weeks to days now we've seen

as little as a day and two now the reason is they want to get to the payday faster after they've done all the brought in all the data they need taken all the data out they want which happens probably in the first 20 minutes then they start looking for your backup platform because they know that if they attack that and successfully remove it from your environment you have no chance of recovering from this event so they're looking for in fact there's malware written specifically to Target certain Legacy infrastructure components that support these systems so they're very good at compromising these backups if they can attack the backup blob itself they'll try to attack the clock or the

time source and then try to spoof time and tell the system that it's 20 50 instead of 2023 and it'll start expiring your backups pretty rudimentary trick but it does work so there's a lot of vectors out there but then at that point after they've attacked your backup platform if they can get it they will if they can't they'll move on they'll malware and they'll head for the door it leaves our customers and folks asking a ton of questions and and these questions are critical to your ability to recover first off how do you know your backups are safe you better be having air gap in a mutable file system or you're in a really tough position right now

hopefully you're doing off-site archiving as well number two how do you know the blast radius it's really difficult to recover from an attack if you don't know which systems were affected and more specifically what data sets on those systems have been affected you have to have observability if you can't have that level of visibility it makes recovery exceptionally difficult data exfiltration how do you know what your exposure is we have to assume every ransomware attack I've been involved with every single one they exfiltrated the data you have to assume if they have physical access or logical access to a workload they have every piece of data on it they took it they have tools specifically crafted to exfiltrate data

I've seen them even if your perimeter is locked down tight I bet you UDP Port 53 is open and they know that too and they'll Tunnel right through it and they'll get your data out very quickly um how do you know a clean recovery Point how do you recover your workloads if you don't know what clean looks like what or good is it's very difficult to recover if you don't know where good is we don't want you to have to recover more than one time the last thing we want you to do is to spend hours or days recovering only to find out you're right back where you started from I hate to say how often

that happens it happens every single day to somebody somewhere and finally automation how do we automate this how do we test it without the ability to automate and test it's a theoretical process at that point so um I'm now going to kind of switch into based on the cases that I have worked in the last three months or so and Beyond previous to that I'm going to talk about the most common things that are tripping points to recovery you'd be surprised how many folks even folks who think that they have a good Disaster Recovery plan or Playbook when the disaster actually happens they realize that maybe it's not as good as they thought or worse yet they don't

have access to it where is it oh it's on the server that got encrypted we have a problem there so you have to continually manage and update those Play Books you need to actually make them living documents most folks we deal with don't have a set in stone document that they can pull out and start working from it really affects your ability to recover if you don't have a documented plan that you've tested and updated frequently you need to perform these practical tabletop exercises to see if your plans are even remotely close to what will really work in fact as I move forward here I'm going to show some other techniques that will help you in your

ability to test those plans because up until you utilize them that's all they are they're just a plan they aren't actionable they aren't operational until you actually utilize them so really make sure that you spend some time having a good rock solid playbook for a disaster recovery and I I mean everybody a lot of organizations have traditional disaster recoveries if the data center gets flattened but they don't necessarily have cyber event recovery playbooks that is a much different event so really think about what it would take to recover your most critical systems from a cyber event what if they weren't available for two weeks what are you going to do um and this really plays into that as well

it surprises me to this day when it comes time to recovery that even mature organizations don't always have a tiered system of criticality for their workloads meaning what are my crown jewels if they're down we're down the business is down you have to recover your workloads in an order that makes sense you need to recover the most critical assets first and then work your way down the long tail for instance I was involved with a municipality who didn't have this order set the the criticality or the order of their recovery was based on who is yelling the loudest and a municipal government had control over the police department the fire department and the parks and rec and

guess who is yelling the loudest Parks and Rec so their support Personnel were doing everything in their power to get Parks and Rec back up all during that time PD couldn't dispatch and fire department couldn't Dispatch they were having to Route the other County in so that's just one example of know your criticality priorities put your assets in a logical order of criticality so that you can recover them in that order you don't want to you don't want to open up reservations for picnic tables before you enable the PD to dispatch calls that's just one example so really spend some time doing that that alone can really make recovery a lot more um achievable you really need to stay up an overall

best practices in both your infrastructure and firmware everything across the board do not ignore patches don't think that you can install them next quarter or next month test them kick the tires apply all patches all firmware the top three threat vectors or attack vectors for ransomware attacks are phishing known vulnerabilities in RDP so it's about equally mixed through those three attack vectors so not patching your Edge servers it's a disaster right now you have to have a robust and ongoing vulnerability management system so that you're staying ahead of these problems a systems that can't be patched get them behind other security mechanisms put layers of controls in front of them because it's just a ticking Time Bomb

they're coming for them they probably know they're vulnerable before you do so pretty straightforward that's been around for decades we've been telling people this for decades keep up on your patch manager um here's one in my opinion which is the number one problem with ransomware recoveries is the human element don't overlook the human element you have to make sure that you've got Access Control in a way that's logical for your environment one example is I when I worked at Microsoft anybody who ever did administrative functions on a system anywhere in the environment you could not do it from your standard workstation they had what was referred to as a sauce a secure administrator workstation everybody hated them he had to carry

loot two laptops around all the time but it was the only way you could do administrative functionality on a Microsoft asset that's what I'm referring to the human element how many orgs have everybody is an admin equip I mean that's got to go away that's the human element I'm referring to here making sure that your users only have visibility access to the critical infrastructure they need to have access to we need to follow the least privileged access model having flat networks with the everyone group like having access to everything on your environment that's just a recipe for disaster that's what these threat actors are taking advantage of that's why they're so successful so in my opinion

the number one thing that comes up in a recovery effort is that the human element has gotten so far out of control that we can recover but you're probably going to get it attacked again unless you apply some really good hygiene to your practices internally so definitely do not Overlook the human element here's a big one too this one causes a delay almost every single time in every ransomware recovery that I've personally worked with having alternative comms people don't even know how to get a hold of their boss or their senior leader during these attacks they they don't have corporate phones they don't the corporate emails down they didn't take the time to establish out-of-band communication mechanisms

very critical set it up in advance make sure that everybody who's going to be necessary for a cyber recovery has access to the out-of-band communication protocols it I hate to say it might end up being Gmail for your communication I don't care what it is you just establish it in advance don't try to figure it out on day two of the attack that's the worst time to try to establish your out-of-band communication policy so make sure it works train people to know it works try it sometime one day just say hey we're cutting over to alternate comms and then just shut it off don't reply to anything except for on alternative columns there's ways to

force it um another biggie establishing a relationship with a outside security vendor um many ransomware recovery operations are delayed if not slowed down significantly because the customer says this is beyond the scope of our capability to deal with we're going to call an outside security vendor I have news for you you can't call them on Friday at 5 pm and expect them to be there at seven that's not how it works you need to have a pre-existing relationship sometimes even a some sort of fee a deposit down to get them a retainer so that they're interested in working with you the more familiar they are with your environment the better it's going to go so spending a little

time to become familiar with the third party security vendor is in your best interest because it will only expedite if the third party security vendor understands what systems are most critical they already probably have some strategy to help you recover so I can't tell you how important it is to have a good relationship that all also includes relationships with local law enforcement potentially as well make sure that if it applies to you especially if you're in a regulated industry or in a government industry there's organizations such as infoguard where you can snap into their community and you can become part of the community to share intelligence and information with them that's why those organizations exist so having that relationship in

advance is definitely a good idea um I talked about this a little but determining what data is important to you it may not be because of regulatory and compliance issues oftentimes it is that's also something that folks worry about there's a lot of mandates that have breach notification requirements meaning that the fines kick in if you don't notify your users by X days you need to know that in advance if you are if that's one of those rules are applicable to you don't wait until you miss the deadline to find out do the research in advance and find out what your breach notification requirements are so there's a lot of ways that you can get a better understanding of your

sensitive data inside your environment obviously take control over in advance because that's what these Bad actors are coming for they're coming for the data the data is the objective of these guys they hold it data they hold it Ransom or hold it hostage they encrypt it they deny you access to it because right now business is data in most business is that we work with exporting your logs another key thing that most folks aren't doing effectively especially with your most critical assets export those logs have syslog export so you can go back and find out what happened later you may not think it's valuable obviously it becomes real valuable after the attack if you haven't set up an event you don't have

it dump it out to Splunk to logarithm qrate or whatever the case may be make sure your systems can handle web hook capabilities as well also consider security orchestrational automation response tools if you're running on a shoestring budget with lack of Staff consider orchestrational automation every security incident is when analyzed by human analyst the same 10 steps are followed every single time very likely why not automate that why not orchestrate that speed it up enrich your incidents you don't have to have it take action but start collect data until you get a comfort level with the action every single time I would have just gone to the next phase so now I'm going to let my orchestration model do it so

consider that as well here's something that is a newish concept and it's getting more and more attention in the ransomware space in the ransomware recovery space consider isolated infrastructure and where this comes into play is that with our with Next Generation firewalls this is becoming easier and easier for instance at rubric we have capability to be able to orchestrate applications for Dr or isolated recovery capability having the ability to fail your application over to a completely isolated recovery zone gives you the ability to recover your production while still preserving the evidence if you are asked to preserve that evidence for an investigation or even if your security team wants to run forensics on it so we do this all the

time you push the images that you are infected to isolation so they can't spread into your environment it enables you to recover your production environment seriously consider it and then also bring those isolation Zone based recovery free plans into your existing playbooks because they can really benefit you in fact you might find that you need to rewrite your playbooks because of your newfound capability to isolate hosts and when I say isolate my personal hypervisor infrastructure has three networks that I can move assets to I've got my production I've got development and I've got isolation production as you would think full-blown production Network my development Network though no lateral movement it can't get to prod it

can but my devops team and security team can get there my ISO Zone separate infrastructure it's a black hole if I push a device there it's not getting anywhere so I use that Network all the time for threat intelligence gathering so consider isolated environments in your environment it gives you a lot of options for recovery okay I'm I'm not going to go through all this because this is some protecting I will share this deck out there's some really common Tech protecting techniques for protection methodology against the most common vectors since they cut me down to 25 minutes from 55 minutes I'm gonna just skip over this but this will be in the deck that gets sent out

basically just talking about the most common techniques putting a little bit of statistics around it giving you a few ideas about common protection methodologies for these most common attack vectors RDP don't even call it remote desktop protocol start calling it ransomware deployment protocol because ultimately that's what it is if you've got RDP on your perimeter they know about it um Everybody stood up RDP at the beginning of covid because they didn't know how they're going to add men their servers guess what they're still there they didn't go away scan for them look for them um also steps to secure RDP obviously pretty straightforward but uh vulnerable public facing endpoints this is right up there with phishing people have devices

sitting on their Edge that have common vulnerabilities that haven't been patched once again I talked about your patching Cadence don't let them sit on the edge unpatched it's just a Sitting Duck it's gonna eventually get compromised make sure that your Enterprise data protection solution has air gap and immutability at a minimum at a bare minimum if you don't have immutability and air gap basically your backup arrays are browseable mountable and tamperable and editable don't let Bad actors edit your backup infrastructure make sure there's protections against premature expiration too because it happens all the time make sure you have a bunker in a box make sure you have end-to-end encryption everywhere MFA everywhere if you can do one thing when

you leave here ask your admins do we have multi-factor authentication on every device on every asset inside of our organization if the answer is not yes make it Bes find a way to implement MFA everywhere that eliminates 99.9 of all attack vectors right there you're going to really mean a much better position simply by enforcing MFA across the board we're moving into the next Frontier in data security so you really have to have some of this functionality Without A system that is monitoring your backups for anomalies they will happen to your backup infrastructure it's really useful if your backup infrastructure can report to you and tell you that it has experienced an anomaly otherwise you're going to

have to find it yourself perhaps even having a methodology to do threat hunting within your backup archive would be useful rubric has that in our platform if you're interested feel free to come chat with us about it but um it all comes down to zero trust architecture methodology so I've kind of come to the top of the hour I'm technically supposed to stop I don't hear anybody yelling at me so I'm not going to until they do I guess but ultimately I put in a ton of really useful all right I said and he did I put a ton of really useful links here at the very end of this the last page I I highly

suggest spending a little time if ransomware and ransomware Recovery is your concern these links are extremely useful so much free open source threat intelligence out there take advantage of it sign up for CSO alerts they may you may think it's annoying getting all those emails but someday you're going to get one go oh my God that's for us that's the Nugget I've been waiting for this is the one alert that's going to save my butt tomorrow so thanks a lot everybody if you want have any follow-up questions I'll be around otherwise thanks for having us