Backup ≠ Cyber Recovery

I almost did we'll give it a couple more minutes we'll get started here folks in just one minute it is straight up uh we're one minute to it you know what we're gonna go ahead they're recording this so anybody who's not here might have the pleasure of uh tuning in later my talk track today is backup does not equal cyber recovery I'm going to talk more about that later but that's going to be a recurring theme of my conversation as we walk through the slide deck um oops not that one a little bit of introduction um my name is Rich Iker I'm an engineer at rubric in our specialty engineering organization my focus has been on our security overlay products that sit on top of our already incredibly secure core platform I've been doing data security quite a while I've been with rubric two years in a couple months but I've been doing data security for a long time about 28 years now um up out of the Seattle Market I've worked for some of the folks you may suspect I've worked for Microsoft T-Mobile Expedia and the likes I've done intrusion detection systems I've worked on IDs IPS vpns vulnerability assessment is where I've spent a bulk of my time in the last 15 years at a very large scale quite frankly um there were times where I was managing up to 6 million assets at one time so that were scanned on a continual basis literally and we never stopped so I did things at massive scale so I've been in the trenches where a lot of you folks will be working and are working today so I understand the trials and tribulations of what can be thrown at you as you're working to secure and work on these systems I'm very passionate about data security feel free to reach out if you'd like more information or have questions after the talk I'll be around connect with me on LinkedIn happy to connect and continue the conversation offline as well um once again though I'm going to be talking about backup not equaling cyber security recovery cyber recovery and I'm just going to let that hang for a minute I mean what does that even mean um it's a strange statement but I want to talk a little bit about it traditionally Enterprise data protection Enterprise data backup has been in the realm of I.T operations for the last several decades probably 40 years quite frankly nothing much has changed in that arena in the last 40 years until recently when companies are re-envisioning how Enterprise data protection should and can work one of those companies is rubric we're a Pioneer in that we've kind of turned the market on its ear in the last several years with some of the capabilities we're bringing to Market that really start to differentiate between a traditional recovery and a cyber recovery from a cyber event recovering a virtual machine is easy recovering a database is easy that's the realm of traditional Enterprise data backup you you'll get a request I need a file I lost a file I lost a server occasionally you'll recover a VM maybe a database that's not cyber recovery that's just traditional backup and Recovery cyber recovery is when you have a cyber event a destructive data event and you need to recover your systems um accordingly and there's a lot of information you need to be able to do that successfully it's not just as simple as recovering a VM we're good now we'll talk more about it as we move through the conversation um the current threat landscape obviously is not good and it's not getting better um on the left there the top activities group that's from this week by the way these numbers are fresh right off the press so as you can see groups like alpha alpha V black bass to lock bit they're not slowing down in fact quite the opposite they're speeding up um top country United States big surprise we have a lot of activity a lot of infrastructure to attack over here as well and a lot of you know economic advantage to attack I put some useful links on the right hand side there my right um these are it's it can be very challenging to get good open source intelligence threat intelligence these are all Twitter feeds that if you follow in fact I almost suggest creating a Twitter feed just to follow some threat intelligence feeds because quite frankly when you do it'll start to take over your feed I'd hate to have you find out the first time first indication of trouble in your organization you don't want to find it on Twitter but you want to monitor it to find out what's going on you'll also get very rich and robust threat intelligence these security researchers are very open and honest about their threat and telling their research I highly suggest following them when you can because they are keeping track of new and emerging threats and trends that you've seen your Enviro that they're seeing in your geolocation so I can't tell you how useful it can be to follow some of those Twitter groups um all right obviously you know these sectors are just getting pummeled look at the health care and the public health 210 incidents um I mean across all verticals are being attacked but as you can see critical manufacturing government I.T shops Health Care they're just kidding brutally attached just constantly and the top threat actors lockbit Alpha V Black Cat And Hive I mean these folks are not slowing down so you really need to get as much threat intelligence as you humanly can against them right now when when there's a ransomware attack and I work closely with our ransomware response team at rubric I should have mentioned that I guess um rubric has a ransomware response team out of our 5 000 plus customers our customers get attacked as well but the difference is our customers have support for them we have a team dedicated to them I work with this team frequently due to my experience in threat hunting right now what we're seeing across the board is that every one of these attacks is incorporating both encryption event and an exfiltration event so we have to really be cognizant of the fact that they're not just locking the data they're just not denying our access to the data they're taking it to use as leverage so a double extortion basically they're storing you for the encrypted data they're extorting you with the threat of leaking your data on the public or on the dark web and I'll even take it to a Next Step tertiary extortion they're actually then turning their attention to the folks that they that are in your data your partners your customers your employees they might start attacking them or calling them we've heard reports that folks whose data lost data they're phoning the people they're calling the people whose data they stole to extort them now so where does this end it's not going to end anytime soon we have to take better preparation the state of data security is bad I hate to say across the board we can all do so much better 98 of it shops out there have indicated that they have faced an attack um so it's I don't need to even really go over some of these statistics you've all heard this it's on CNN It's On The Wall Street Journal it's everywhere it's coming from the board down now so the first time in my career people from sea level to senior leadership are coming to us they want to know what is the status and the reason is these statistics they're losing Revenue they're losing money they're losing reputation and they can't continue to do this it's affecting the stock prices this is affecting the shareholder value or the personal value of the company and it's really something that has to be addressed so it comes down to the simple fact that businesses are absolutely under attack and we live in an assumed breach world and if these Bad actors get into your environment and they get your data they really got you but if they get in and get your data and your backups they really have you they have you in a bad way in fact this suddenly can be a lights out scenario for some businesses because if you don't have your backup data to recover from it could easily be a lights out scenario it could take you months to rebuild um we have to worry about if our data is an easy target in a traditional system Enterprise data protection system you might have multiple systems from your disk your sand your disk based backup your backup servers your backup proxies these are oftentimes all connected to your ad infrastructure all running on Windows infrastructure multi-factor authentication is not widely implemented and certainly not widely enforced basically this is becoming a recipe for disaster right now we're seeing this get attacked again and again and again in fact there's ransomware on malware specifically designed to find and attack this type of infrastructure setup so if you are running one of them the due diligence is on your shoulders now to watch that thing like a hawk if you don't have multi-factor authentication on these get it get it fast because they're coming for you so ultimately the complexity level you add in the next level of complexity which is your Cloud infrastructure and most of our customers at rubric not most many are multi-cloud customers they're in all three major players AWS gcp and Azure so the level of complexity that this Cloud infrastructure is bringing to the table further complicates your security model so if you've already got this type of scenario in your environment you add in your cloud workloads and then all of a sudden the level of complexity is almost out of control you might need multiple full-time employees just to keep control over this environment and of course this is where you get the uh you know the information you need not always being available how do you know what to recover if you're attacked in these environments if you don't have Rich visibility into your data how do you know what sensitive data could have been involved in the X in the breach and it was exfiltrated and how do you know that you're when you're recovering that you're not just bringing the malware or ransomware back into your environment where it re-detonates in your right back where you started from that's the last thing we want to happen because now as I said the c-suite and the board is watching and they want to know questions like are we good can we recover how long is it going to take these are questions that can be extremely difficult to answer and they want these answers and they want them soon so ultimately let's talk a little bit about the anatomy of these attacks and then I'll get into some of the common failing points in recovery but before I do I do want to talk about this anatomy of a ransomware attack let me just have one set um as I say rubric has a ransomware response team we have helped hundreds of clients to this point recover from ransomware I have personally been involved in 18 in the last three months and that's just the ones I've been involved in so we've learned a ton of really useful information when working with our customers as they recover from these cyber events so I want to just cover the anatomy of these attacks as we see it first and foremost they get in it just takes one mistake regardless of the amount of money you've spent on protective methodology it just takes one mistake it could be uh Executive Admin who clicks the wrong email and your 100 million dollar xdr investment just went out the window the first thing they do is they stand up command and control infrastructure they do this to bring data in and to take data out the data they're bringing in is the Hostile malware the data they're bringing out is your data they're specifically looking for any data to exfiltrate to bullet point number two they really love financial information they especially love your cyber insurance policy if they can find that they know exactly what they can expect to extort out of you um of course during this time they're also trying to increase their footprint both vertically and horizontally they're trying to spread out and they're trying to move their access up all during this time I mean a lot of activity in the first bit and also they're not sticking around as long as they used to either dwell times have come down from on average of weeks to days now we've seen as little as a day and two now the reason is they want to get to the payday faster after they've done all the brought in all the data they need taken all the data out they want which happens probably in the first 20 minutes then they start looking for your backup platform because they know that if they attack that and successfully remove it from your environment you have no chance of recovering from this event so they're looking for in fact there's malware written specifically to Target certain Legacy infrastructure components that support these systems so they're very good at compromising these backups if they can attack the backup blob itself they'll try to attack the clock or the time source and then try to spoof time and tell the system that it's 20 50 instead of 2023 and it'll start expiring your backups pretty rudimentary trick but it does work so there's a lot of vectors out there but then at that point after they've attacked your backup platform if they can get it they will if they can't they'll move on they'll malware and they'll head for the door it leaves our customers and folks asking a ton of questions and and these questions are critical to your ability to recover first off how do you know your backups are safe you better be having air gap in a mutable file system or you're in a really tough position right now hopefully you're doing off-site archiving as well number two how do you know the blast radius it's really difficult to recover from an attack if you don't know which systems were affected and more specifically what data sets on those systems have been affected you have to have observability if you can't have that level of visibility it makes recovery exceptionally difficult data exfiltration how do you know what your exposure is we have to assume every ransomware attack I've been involved with every single one they exfiltrated the data you have to assume if they have physical access or logical access to a workload they have every piece of data on it they took it they have tools specifically crafted to exfiltrate data I've seen them even if your perimeter is locked down tight I bet you UDP Port 53 is open and they know that too and they'll Tunnel right through it and they'll get your data out very quickly um how do you know a clean recovery Point how do you recover your workloads if you don't know what clean looks like what or good is it's very difficult to recover if you don't know where good is we don't want you to have to recover more than one time the last thing we want you to do is to spend hours or days recovering only to find out you're right back where you started from I hate to say how often that happens it happens every single day to somebody somewhere and finally automation how do we automate this how do we test it without the ability to automate and test it's a theoretical process at that point so um I'm now going to kind of switch into based on the cases that I have worked in the last three months or so and Beyond previous to that I'm going to talk about the most common things that are tripping points to recovery you'd be surprised how many folks even folks who think that they have a good Disaster Recovery plan or Playbook when the disaster actually happens they realize that maybe it's not as good as they thought or worse yet they don't have access to it where is it oh it's on the server that got encrypted we have a problem there so you have to continually manage and update those Play Books you need to actually make them living documents most folks we deal with don't have a set in stone document that they can pull out and start working from it really affects your ability to recover if you don't have a documented plan that you've tested and updated frequently you need to perform these practical tabletop exercises to see if your plans are even remotely close to what will really work in fact as I move forward here I'm going to show some other techniques that will help you in your ability to test those plans because up until you utilize them that's all they are they're just a plan they aren't actionable they aren't operational until you actually utilize them so really make sure that you spend some time having a good rock solid playbook for a disaster recovery and I I mean everybody a lot of organizations have traditional disaster recoveries if the data center gets flattened but they don't necessarily have cyber event recovery playbooks that is a much different event so really think about what it would take to recover your most critical systems from a cyber event what if they weren't available for two weeks what are you going to do um and this really plays into that as well it surprises me to this day when it comes time to recovery that even mature organizations don't always have a tiered system of criticality for their workloads meaning what are my crown jewels if they're down we're down the business is down you have to recover your workloads in an order that makes sense you need to recover the most critical assets first and then work your way down the long tail for instance I was involved with a municipality who didn't have this order set the the criticality or the order of their recovery was based on who is yelling the loudest and a municipal government had control over the police department the fire department and the parks and rec and guess who is yelling the loudest Parks and Rec so their support Personnel were doing everything in their power to get Parks and Rec back up all during that time PD couldn't dispatch and fire department couldn't Dispatch they were having to Route the other County in so that's just one example of know your criticality priorities put your assets in a logical order of criticality so that you can recover them in that order you don't want to you don't want to open up reservations for picnic tables before you enable the PD to dispatch calls that's just one example so really spend some time doing that that alone can really make recovery a lot more um achievable you really need to stay up an overall best practices in both your infrastructure and firmware everything across the board do not ignore patches don't think that you can install them next quarter or next month test them kick the tires apply all patches all firmware the top three threat vectors or attack vectors for ransomware attacks are phishing known vulnerabilities in RDP so it's about equally mixed through those three attack vectors so not patching your Edge servers it's a disaster right now you have to have a robust and ongoing vulnerability management system so that you're staying ahead of these problems a systems that can't be patched get them behind other security mechanisms put layers of controls in front of them because it's just a ticking Time Bomb they're coming for them they probably know they're vulnerable before you do so pretty straightforward that's been around for decades we've been telling people this for decades keep up on your patch manager um here's one in my opinion which is the number one problem with ransomware recoveries is the human element don't overlook the human element you have to make sure that you've got Access Control in a way that's logical for your environment one example is I when I worked at Microsoft anybody who ever did administrative functions on a system anywhere in the environment you could not do it from your standard workstation they had what was referred to as a sauce a secure administrator workstation everybody hated them he had to carry loot two laptops around all the time but it was the only way you could do administrative functionality on a Microsoft asset that's what I'm referring to the human element how many orgs have everybody is an admin equip I mean that's got to go away that's the human element I'm referring to here makin