Ransomware and Backups: A Multi-Layered Defense Strategy

thank you for attending bside San Francisco 2024 our speaker today is presenting and his name is Amal from cyber resil he's the Cyber resilience leadership at verites Technologies before we get started what I would like to do is let you know that we would love your participation um so sli.do um is what we're using to ask questions we will open for question and answer towards the end of the presentation you can also there's QR C codes outside of every theater on the signs if you don't if you didn't scan it yet um you can use bsides SF or uh.org qna my name is nicolina and it's a q like Quebec nicolina Alpha so no Amper

sand is actually the letter N okay and um all right sir take it away thank you so much thank you all right um well thank you for being here uh it's the second day and welcome to the last few talks at besides San Francisco as the title suggests this is a presentation on ransomware and backups so if anything I would like you to take one word away from this talk is that it's a multi-layered defense strategy multi-layered and throughout this talk we'll sort of know why that multi-layer word it's so important so yeah let's get started um few years ago or I should say actually many many years ago uh companies used to perform backups so

that if there was let's say a natural disaster or some uh CRC error data Corruptions things like that or if someone maybe just deleted a file by mistake then you could restore it but um not anymore sadly not anymore most the most of the time today we back up things so that we could possibly restore from uh ransomware attack so what we are going to do today is we'll take a quick look at the ransomware landscape we'll look at uh some of the attacks and some of the simplified version of ransomware attacks uh and have draw a lot of parallels between U Network endpoint security as well uh to backups so what I mean is essentially uh in your typical

Network word you would have a endpoint admin or a network admin and they do a certain task before an attack so they would do vulnerability scanning they would do posture management to make it configuration is good and what we'll do is we'll see what as backup administrators what backup administrators can do before a ransomware attack similarly uh when the ransomware is in the darant phase you would have your uh threat Hunters trying to hunt for uh dormant malware in your network on your endpoints and whatnot and similar uh in parallel to that we would also see what um backup administrators how can they hunt uh in backups based on the data that they have in addition to what

all your endpoint folks and network folks and your I folks are doing uh and last but not least we'll go we I have a checklist that I have from uh from someone I'll just give a link to that 30 60 90day checklist that uh would help any organization in their uh in their backup and Recovery effort so let's look at uh Ransom word this is from the most recent FBI crime report it has data on a lot of things in addition to ransomware but what this states that um about in 2023 which is the most recent version of the report for the entire year last year there was about 74% increase in the money lost due to

ransomware so that's a pretty staggering amount there was also a 18% increase in the complaints that they got on ransomware so the way this report works is that once uh there is a data breach a lot of organizations and again depending on which uh which uh vertical you are in you have to report um that such and such incident happen and this report takes that data and um compiles it so one of the things you could see is that the the increase in the dollar amount is a lot more than the increase in the number of complaints so I think that possibly means that uh ransomware uh gangs or Ransom where is going towards more

lucrative targets where they can get the most bang for their Buck um as the number of incidences are increasing slower but the money lost is increasing dramatically so from the same report this is the chart for which verticals were affected so uh you had your your Healthcare and public health affected the most followed by uh critical man ufacturing followed by government and yada yada yada so it's a pretty nice report I think there was a I'll share the slides and the link is in the slides uh it's a it's a good read I was I think hoping uh the MGM attacks uh that would put uh some sort of Dent and Healthcare would not be number one but uh maybe FBI

included all the casino machines in critical infrastructure so I don't know everyone knows was about the MGM data breach right I mean that was like a very big data breach where all the casino machines all the hotels MGM um Caesars and everything was offline for a long time so that's what I was referring to um no surprise here again this is from the same report it's the top malware um and malware gangs that that they are reporting so um then I I I said okay this is at the FBI level at the federal level since besides is in San Francisco California let's take look at some of the local numbers and this data is a

little bit more uh interesting because the FBI report already gives you some pre-compiled data but the California data security breach uh website they give you raw data so I think that was really nice so that you can do a bunch of calulations on it and get to your own conclusions so this is the raw data as you can see in 2024 which is the last bar here we are already at a at a level for the entire year which where we were 5 years ago so that is that some of the conclusions that I drew from the from the raw data at uh on the California data security breaches is that there was in California

41% increase in the number of breaches so pretty similar to what we saw at the federal level 38% increases uh 38% increase in repeated breaches on the same business so what that means is 38% of uh the customers or the businesses they said that they were attacked or data was infiltrated Data breached happened more than once uh there was a 45% decrease in the time between the first and the second breach at the same business so if you compare the data from 2022 and you compare it with 2023 then uh attackers are basically getting quicker at having multiple uh breaches at the at the same business and uh well this was sort of my not really favorite but there was an

organization which got breached 11 times in 2023 so um I mean although I hear some giggles and but like when you look at the California data breach report it has real names real uh organizations and I was like oh that is my insurance company oh that is what I have a device in my anyway I won't go into details but these are compan companies that I personally work with every day or I see on Highway 101 so why is ransomware having so such a success even after 20 25 years I mean ransomware is not a New Concept right um and I think to find that out let's quickly look at the different techniques and tactics that

ransomware deploys so I think how many of you have heard of the miter attack Matrix hopefully everyone so good so miter attack is um is a set of techniques and tactics that they constantly update that's a very important word constantly update uh which ma malware or ransomware deploys and U we are not going to read through all of these but there are various different buckets here of what happens before the attack or what happens during lateral movement credential thft so on and so forth and your xdr vendor I mean in my past life I used to work for xdr vendor so a lot all of the xdr vendors or at least what my team used to do is

we used to create detections for our product so we used to say oh this technique this tactic our uh we have to have a detection this technique this tactic we have to have a detection and that's how all the xdr vendors say that okay we have let's say 90% coverage for miter attack what they mean is they can detect uh malware if it uses any of these uh techniques and tactics but today what we are going to do is we're not going to go into the miter attack uh um in detail what I've done is I've basically just classified it into four broad buckets one the first bucket the green bucket is the prevention bucket

this is basically pre-attack and what we'll do is we'll see what uh organizations can do uh before an attack in uh with respect to backups again there are a lot of things you could do with respect to endpoints Network and all all those things but what we are talking here is specifically with respect to backups what organizations can do in the dormant mode where um malware is in your organization and is trying to move laterally and your folks are also trying to do threat hunt at the same time and uh also during the detonation phase so let's get assorted so first is prevention so this is where basically as a backup administrator or as a uh as a department responsible for

uh securing backing up data you would create strategies before a data breach happens um this is a pretty common uh thing in the backup world I think to pass your cissp you must have known all of these terms but what what you have here is on the left is you have your last backup done the left uh box and um then basically at some point there is a there is an incidence and the time between your last backup and your incidence is your RPO uh recovery Point objective then between your um last event and you go to recovery what you have is your RTO from your recovery to completely testing and building uh getting your systems online is your wrt

and then you have your normal operations continue so I think one of the most important thing you could do as a as organization that is responsible for um backups is see where you fall in your rpu RTO what are these values for your organization what are what is really realistic for you um you can start with data classification there are many methods to classify data some military has those type of techniques commercial just do four techniques but find out what is your uh way to classify data and then use one of the existing um existing contingency planning techniques so nist 853 has a pretty good um pretty good advice on how you should do your

contingency planning basically identifying backups uh comparing backup types using uh uh some sort of a backup solution and things like that so by all means try to modify it put your secret sauce in it but also look at some of the very uh easy I mean standards that have already been done by some other folks who have thought about it for a long time and Implement those standards this is again pretty common in the backup world is what is your backup strategy 321 is a pretty famous one so basically you have your three copies of backup in two different Medias and one is offsite now there is the asteris on one because uh it used to be that this

one is offsite copy now with Cloud there is a debate if cloud is considered as offside copy or since it can be accessed from inside is it really offsite or not there is a lot of discussion on this topic but U yeah uh try to see what is your strategy there are other variants like 3211 and the last the additional one is for um for a disconnected copy or 32110 and the last zero is for uh no errors during your backup or there is 432 so there are many different strategies but again in the prevention phase this is an important decision that you would make um and last but not least nist again has a example of um an

example organization so your organization would not look like this it would be different but this is just an example organization and again the link is below where nist tells that okay in these situations use worm data in these situations use encryption in these situations use uh Cloud backups and things like that so take a look at uh these examples and what are the best practices and recommendations and that generally has uh would would help greatly last but not least in the prevention category is testing recovery and this is my personal favorite because no amount of planning or using backing up or using different type of software would be useful unless you test your recovery the last thing that a

backup administrator or backup organization wants is when it comes time for Recovery there is some error or there is something that happens that says oops cannot recover and that is what I think everyone in this room or in in the backup world dreads that when it comes time for Recovery it doesn't work so having a very good test plan and not just test plan for recovering one or two workloads but uh in in real life situations you would do bulk recovery if you have bulk problem so test your recovery in bulk so these these were some of the uh things that you could do in the in in the prevention phase in the dormant phase now things get a little

interesting so this is when a malware is dormant in your network but it has not exposed itself so thread actors are in stealth mode inside your network there is no data loss yet no encryption yet what thread actors are actually trying to do is they are trying to gain privileges they are trying to evade your defenses they are trying to elevate privileges or uh move laterally access credentials things like that and more and more recently they are also trying to Target your backups because it's think about it I mean if you had a magic wand and you could restore all of your data instantaneously then it really takes out the power out of the ransomware because you can really

restore very quickly so what ransomware gangs are trying to do is they're trying to find where your backups are kept do you do your own backups are they just in one of the shared drives or do some administrators who has access to those share drives do you use uh a vendor solution which solution are you using can I as a malware writer let's say corrupt those backups and things like that so more and more that is happening in this phase where your backups are targeted and also your backups can provide malware a very or or thread actors a very good blueprint or of where your crown jewels are so the the idea here is you back up mostly I

mean you backup your crown jewels if somehow by credential theft or by any other means I can get into your backup software then I can see the workloads that you're backing up and then I know oh I should laterally move to these targets because these are the ones that you are backing up so as Defenders what can we do well one of the things is we could since we know that they are targeting our backup software we could use indicator of attacks on the backup software so essentially have uh some sort of a user behavior analysis on the use of the backup software so either your vendor would provide something out of the box or what you

could do is you could have all your access logs for your backup software in a ubaa type of solution and then do some data analysis there is like was this user's Behavior anom anomalous this user never really adds any other users why did he add it all of a sudden or this user never disables any backups or we are you know we have to have a big meeting before we disable anything but this you administrator disabled something so was this the administrator or someone has got uh his credentials his or her credentials so I think user Behavior Analysis would go a long way uh there is also a talk of using your backup software as a honey poot inside

of your network so everyone knows the concept of a honey pot right like you um you intentionally keep a badly configured uh backup software backup server in your organization you put some breadcrumbs around it breadcrumbs is also a pretty technical term just Google around it so you the attacker you are sort of guiding the attacker inside of your organization with the breadcrumbs in active directory or things like that to the backup software to your fake backup software that's where you are monitor ing everything so this sort of becomes a part of your threat hunt where you're trying to uh detect um dormant mode malicious um users in your organization uh one of the other things

that you could do is uh well you have backup data right you could scan your backup data just using malware scanners so now this really depends on how the data is stored if you if it is stored encrypted if it is is D duplicated and things like that then you would have to decrypt it you would have to basically have it in a state where malware scanners can actually work on them so um if if your vendor provides this functionality it's good uh the only con of this process it is pretty slow as compared to the number of or the terabytes of backups that you have if you start scanning each and every backup it could become slow so

that's that's the downside of it another thing you could do just like IAS or indicator of attacks you could look for indicator of compromises inside of your backup data now just as your um endpoint vendors they do Shu 56 or the context triggered pie piew hash matching to find where is malware where is dorant mode malware which is basically malware that has not run yet or it is waiting for uh command and control or some thing to tell it that okay now it's a go time go start encrypting go time do your destruction we are still in that dormant mode so uh you could also uh do the same thing on your data backups as in uh look

for uh malware hashes and other indicator of compromises in your uh backups where do you get the CTP and all these are pretty interesting things these are different types of hashes that uh that can identify exactly a file or identify almost identical file so yeah so where where do you like get these ioc's I think that's pretty easy every FBI advisory every every cisa advisory has these ioc's you can get them from those advisories you can get them from a lot of open- source threat intelligence sources um and again I've just put a lot of links here I'm not going to go through all of these but there are a lot of open-source threat intelligence um feeds almost that you

can uh use to get those ioc's and last but not least you can use uh commercial vendors as well to get ioc feeds but the main thing is you can do this type of threat hunt on your uh backed up data and again this is from the perspective of a backup administrator your endpoint Network and other AD administrators would already be doing some different type of threat hunt uh last but not least uh Yara I think how many of you have heard of Yara this is uh oh cool so this is like a language where it can um it can analyze files and it can tell you if that file has some malicious behavior inside of it or not and everyone cisa

and everyone recommends that you use Yara for uh your threat Hunts uh an example of things that it can detect is basically a lot of malware and ransomware uh first uh checks if they are running in a sandbox if a if a researcher is trying to analyze it and if it thinks it is running in a sandbox then it would not run essentially it would not uh do the bad behavior so there are U again lot of GitHub repos here there are lot lot of Open Source Yara rules that can identify such uh binaries or such files that have anti-debugging techniques in them and a normal file normal executable or any type of file should not have any

anti-debugging techniques built into it um a normal file should not have an exploit kit into it exploit kid is essentially almost like libraries that malware use reusable libraries uh to um do whatever it needs to do so even malware just like the software we write it's not written from scratch right it just assembled you take a library from here you take library from there you assemble it so there are rules uh to find these exploit kits There are rules to find various cves or vulnerabilities web shells the list is endless so again uh take um go to these GitHub rep and see what all uh what are the things that you could run on your backup data to

detect dormant mode malware so uh let's switch gears to from dant to detonation phase so what we are trying to do is we are trying to base on our data backups see where um malware is so in this phase what has happened now is thread actors are now actively and en crypting data actively deleting data actively infiltrating data and they may or may not have disclosed themselves but the data is changing and um well before we go into this one of the questions that would come into mind is why would I need to detect malware in this phase because wouldn't my network admin or my endpoint admin already tell us that hey we have an infection because there is a

notice on his screen that hey you have been infected this is the ransom so there are two reasons why as a backup administrator from the data perspective it would help you one is uh you could have rules where you stop backing up data if you see a lot of variance in data so let's let's actually uh jump into it so you could observe anomalies in the data that is backed up so one of the anomalies that you could see is the data entropy so entr entropy is basically um the randomness of the data low entropy indicates that there are more like regular patterns in the data and high entropy means that your data is highly random so if You observe uh

anomalies or rapid changes in this so for example I'm backing up a workstation uh or server a database kubernetes cluster whatnot but I see a lot of entropy change as compared to my Baseline for the last week last two weeks then I know that data is changing rapidly and I could use that to stop the backups because I know that usually this is not the normal behavior and I don't want to override my backups with you know with bad data or with backups that ransomware has already uh encrypted those files or deleted those files and things like that so A lot of times what happens is um you uh you have a lot of backups but then each backup you see you

already have the a bad data you already have data that malware has touched so you can use this to automatically stop or overriding your older backups or at least you can at least see what's going on before you stop uh your backups other thing is dup ratio this is also very common technique uh when it comes to backups it's essentially storing only uh unique data and then storing just references to that data and by again observing changes in uh this uh particular um what should I say attribute of your data you can come to a conclusion that this data is our data is changing rapidly than it did like a week ago or based on our Baseline and then

that can indicate that there is ransomware trying to do some sort of uh m changes on on on my data other things like file uh access um file attributes and things like that would also suggest the same usually like if the usually if there are certain patterns of access for files and you see that there are those patterns have changed dramatically they would definitely change from backup to backup uh but if they are changed dramatically as compared to your Baseline then again you could say that hey this is something I need to investigate more so uh that these are the things you could do when ransomware is active and it is actively changing modifying deleting and

filtrating your data and as a backup administrator you can uh make sure you're you are not again backing up corrupted data and it also helps in recovery so when it comes time for Recovery you can can uh easily go back and say oh let me see my data which has a low uh change in dup or low change in uh various attributes that we saw so you you can uh quickly identify what is good data versus what is bad data when it comes to restore so well last but not least uh in the recovery phase I think uh what what you do is you basically make sure that all your RTO and rpos are in line with what you had in mind

recovery has a lot of steps in it again I'm not going to go through each of these steps U but these are generally the steps that one follows during recovery and I think the most important uh step here or the most important thing here is if we had done our rehearsals on recovery well in the planning phase in the green phase then this phase should be well I wouldn't say a breeze but should be a lot lot easier if uh if uh recovery uh rehearsal was done well and I mean hopefully if it comes to it you say that okay I did a very good uh my rehearsal paid off and I think that is

what you want at the end of the day is after a malware breach after a malare attack uh when it comes for Recovery you pat yourself on the back and say oh my rehearsal paid off my bulk rehearsal paid off so that is pretty much it I think this is the 30 60 90day checklist that I have and again there is a link at the bottom that you can uh go to and get more information about this 30 60 90 days checklist for like a zero doubt um recovery and and uh yeah that's all I have I would like to thank the bides for having me here and mostly thank all the volunteers because as you know this is a

very volunteer focused event so thank really thank all the volunteers here for their contribution thank [Applause] you thank you so much Jamal for that wonderful presentation it was most most certainly enlightening I want to thank everybody for coming we actually have a special gift just for you I was joking with him earlier that he would only get it if he behaved and he definitely did so thank you uh we want to thank all of our sponsors and actually that was a gift from socket Securities especially um so we're going to open up for Q&A we're going to ask some question there was actually a couple that um that already got posted and if we if you want to um add

any more you you can go to bides sf.org SL q& so the first question is thoughts on hourly snapshots and daily backups Ah that's a um I think it's a very good question it's not an easy one because um yeah I can recommend is take hardly snapshots take a snapshot every half hour but practically when you look at it it it it has to do a little bit with your organization what is the type of data you're backing up and what you have done in the past so usually my answer is as long as your trend is going upwards Asing you are doing better and better let's say you're in your organization you only had a weekly

snapshot let's say then I think the T your target should be okay I want to do this now twice a week instead of just a week if you had it already twice a week you should a aim for maybe even daily and things like that because I don't think like one uh sort of solution fits all and it really depends on the type of workload the amount of data that you have and many many other things so good question great question okay where did change healthc care fail in their backup strategy in regards to their ransomware attack that's a very specific question uh um I did study that you that that youth case a little bit uh possibly not

enough to give a very conclusive answer where a particular organization failed um so sorry I I I I really don't know I don't want to just guess here thank you for that and also could be by proprietary so we won't make you answer um have you seen examples of false positives of anomaly detection and can you speak to the challenge of tuning of anomaly detection MH MH yeah yeah yeah absolutely I mean anomaly detection and again not only in the context of backups but in the context of uh like let's say network security or your xdr alerts there are by definition there will be false positives in anomaly detection the key thing to watch there

is what is the rate of those false positives as in um and is it acceptable to you can you even triage those false positives if the false positives are so much that you pretty much start ignoring them then that is uh something that you would want to take up and see why that is happening there would be false positives no doubt about it um because because anomali is just you know based on some data looking at your past patterns and see that the newer pattern has changed so I mean there could be two reasons for the false positiv one it's uh maybe the implementation of that anomaly is bad or second is genuinely your user uh patterns are changing and

there is really no malware but your user P patterns are changing uh before this you were adding users let's say by uh By Hand by clicking but now you have a script to add them so now users are being added deleted modified more rapidly and that can obviously trigger alarms and anomalies say oh too quickly things are happening but that I would say I mean it's it's on it's on the Bott line of false positive because it really did identify that the pattern has changed so I think that's my yeah that's the awesome thank you so much um we have a couple quick announcements um so we have head shot available I'm not sure if you are

all aware but right outside of the talk area you're welcome to get your head shot and update your LinkedIn also um as a reminder you can support Charities eff Golden Gate Gardens Park and techbridge girls by buying a t-shirt by the coat check on the top floor we look forward to seeing all of you again thank you so much for coming to bside San Francisco and hopefully we'll see you next year all right thank you so much let's give him a great Round of Applause thank you thank you sir