BSidesROC 2025 - Cloud Storage and Ransomware Attacks - Velizar Demirev

attendance that you know of so far uh 220 or something so far we had 3 more than 350 people registered so we'll see who else shows up got yeah I know a lot of folks trickle in you know kids and everything that that might come later in the day good morning we're g to get started with our last Talk of the morning in here so come in have a

seat I would like to introduce vazar deev and he is uh he for the past four years he's worked at Cloud Storage security supporting customers researching security and doing anything that's needed to help keep the company moving and with that I will turn over to Valar thank you oh [Applause] almost forgot we want to give him a a little gift for speaking too here you go oh thank you appreciate it all right uh so let's go ahead and get started here uh today we'll be covering a little bit about uh object storage uh incidents related to anything object storage related in the cloud uh and uh just putting a plan around uh forensics for it uh so real real quick

my name is Bazar uh I went to RIT uh I've worked in uh the cyber security space and just startups in general for a while ever since uh going to RIT uh right now I'm at CSS Cloud Storage security as CTO uh doing just about anything that's needed uh day-to-day uh at CSS we build uh software that is uh used for malware protection as well as uh we're building something new coming up soon uh for the data layer to protect your data in the cloud uh we also have an internal threat lab uh that we recently started called casmer Labs so we're doing a lot of research as well on threats uh in the cloud uh related to

the storage layer so for today uh we're going to break this down into two different parts uh the first is we'll talk a little bit about uh putting together a forensic plan through the lens of uh the nist 161 R3 framework uh and then we'll actually also talk about putting that plan to use in AWS Azure uh and gcp so uh just to get an idea here how many of y'all use object storage so S3 oh wow that's that's quite a few yeah everyone indeed who doesn't these days um one of the things we've noticed is that organ organizations have a lot of Assets in the cloud uh and they have hundreds thousands of buckets so

it's heavily used and a lot of the time it's also multicloud um there's really one uh Golden Rule that I like to uh stick to uh and that's regardless of the objects whatever they may be file types however they you know they get there the golden rule is they should only be accessible by anyone who's authorized is to access them and that's not always so easy uh at the end of the day um over 60% of data now is stored in some type of cloud uh and that number continues to grow year after year um breaches for data about 40% last year according to IBM were cloud-based and based on our own internal research uh through our threat

lab uh over 40% of uh organizations have fallen victim to some type of ransomware attack and we'll talk about it a little bit later on ransomware meaning both uh files that uh create some kind of ransomware attack at the end point but also the uh storage volumes themselves being victim to ransomware so everybody's shifting their focus towards data uh when it comes to attackers and as you can see the the consequences are always in the news uh day after day we see folks uh having some type of breach some type of leak whether it's due to a bucket becoming public accidentally uh due to misconfiguration or it's due to uh some type of malware being uploaded into the

bucket and ending up Downstream on an employees device or customers device so on and so forth It's always in the news you continue to see this time and time again um when we talk about the cloud it's important to remember that uh all of these different platforms have some level of shared responsibility uh where the cloud provider is focusing on the infrastructure so the hardware as well as the software side and you as the customer need to focus on actually configuring the uh uh different software properly as well as focus on what you're actually storing on that infrastructure so it's your responsibility at the end of the day and this is just an example from the AWS side but Azure gcp all of

these different platforms have something similar where at the end of the day you're responsible for knowing what you're storing in the cloud all right so uh let's dig into this a little bit um the nist framework the 861 R3 framework is focused on forensics um before we do so though I do want to get a little bit into what types of threats to expect uh when you're dealing with cloud storage and it mainly focuses on three different areas the first is U malware uploads so when it comes to malware usually you think of it as downloading some malware on your computer but a lot of the time when it comes to cloud storage there's that

Downstream risk even if even if a file is not executable in an S3 bucket for example the downstream risk that it creates is is the real threat if that file ends up on your machine your customer machine your partner's machine whoever it may be in your organization or outside of your organization that's accessing the file Downstream that's a big problem a lot of folks say well I have endpoint protection on my devices why do I need to worry about this uh well not every single uh piece of or I should say not every single vendor anti virus vendor is going to be able to catch uh every single threat there's no guarantee as well that everyone that

accesses these files has some type of protection on their devices so the key is you want to scan for malware uh at the source as it's uploaded um so that's number one two is misconfiguration so especially if you have a lot of buckets like we've seen here everybody it looks like has something somewhere in a bucket in the cloud stored uh misconfiguration is really easy to uh fall victim to um we're all human at the end of the day and human error is Big when it comes to cloud storage is and that's what we found at the end of the day so making sure that whatever you're storing is properly uh uh set up properly

configured both at the object level as well as the storage volume level is something that you'll need to make sure uh to focus on uh and then last but not least uh ransomware in Rion so like we talked about you should expect ransomware both at the uh object level the device level uh as well as the storage volume level uh so those are just the three things I want you all to keep in mind as we go through this today all right so the nest framework uh it's broken down into two different pieces which we'll talk about here uh the first part of it focuses on Preparation uh before an incident occurs and the second part of it focuses on the

incident response itself after an incident occurs so uh it's split out into these two sides on the preparation side it's broken down into three pieces uh govern identify protect as well as on the incident response side it's detection response recovery and I should know by the way the revision three of this NST framework uh actually released this cyber security 2.0 uh framework so this is this is the latest as of last year so um let's talk a little bit about preparation when it comes to this uh this framework here um the first step of it is governance and that's really going to vary uh based on each organization based on each team based on you as a

team member in an organization uh you are responsible for setting everything related to uh priorities in terms of incident respons resp uh in terms of what your tolerance is for a threat and as well communication as far as internal communication to team members uh whether it's your security team your legal team your executive team whoever it may be as well as external communication to your customers and whoever you're you're working with externally so governance is unique um they mention it very specifically but but keep in mind it's going to be unique to your organization it's just something that you you and your organization are going to have to be responsible for um identify so keeping an inventory of all

of your assets all of your buckets all of your storage volumes that you have in the cloud is going to be crucial in order to prepare for some type of incident um not everybody has uh an idea of all of the storage volumes that they have we run into it a lot of the time where a customer just doesn't know that they have some level or some set of buckets or storage volumes in a specific region in AWS for example uh and they'll discover them and they'll say well that's not supposed to be there so having proper inventory is a crucial part of planning and then uh protect so implementing defenses so uh proper configuration uh goes Miles and Miles um

things like proper Access Control having encryption enabled it's it's really crucial at the end of the day as part of that preparation um as far as incident response goes so after an incident occurs um the first piece to this uh framework is uh detect so you want to be able to have something in place to monitor for anomalies um usually a seam or sore type tool like Splunk for example is great for that because you can aggregate all your logs into it and alert for uh specific activity uh when a incident is detected you'll want to isolate and eradicate so the response piece of this uh is broken down into into two two different sides there's the

action that you take uh to isolate the threat to stop it in its tracks but in the framework itself it states that you need to have clear communication so this kind of goes back to that governance side of you want to make sure that you're communicating with your team internally and externally as well to your customers and then recovery um the last part is like we all uh probably know uh backups making sure that you have proper backups and are prepared for an incident uh making sure that if you do back up that the data uh uh you you verify the Integrity of the data and as part of that uh recovery process the second part I should say because I would

break this down into two different parts uh the second part is you want to perform some type of analysis on what happened to learn from it figure out how it happened so on and so forth so we'll get into it a little bit more here um when it comes to preparation uh it's broken down again into these two different sides identification uh of threats it could be things like misconfiguration so a public bucket uh a bucket that might not be public but the objects in in them some of them are configured to be public so on and so forth uh permissions overly permissive actions so a user may have permissions that they may not necessarily need at

the end of the day uh no encryption either at rest or in transit and then uh your backups being vulnerable so um we'll get into it a little bit more as well on the backup side but again keep in mind ransomware could be a big issue both at the object level and the backup level as far as protection goes um and and to protect yourself against threats as part of the preparation process you you want to have proper Access Control you want to encrypt and you want to have backups that folks can't change so um let's talk about this some more here when it comes to the detection like we talked about you want to have a

seam tool for alerting um and it's really a matter of is someone uh is someone doing what what they're not supposed to be doing so uh is a user account uh performing some type of action that they're not supposed to whether it's during the daytime hours or nighttime hours or whenever um along with that that ties into monitoring for behavior and misconfigurations a lot of it will get into here um I'll just Skip by that there we go um when it comes to uh after a uh uh an incident occurs it's broken down into four pieces uh like we talked about so isolation if there is something that occurs uh you'll want to go through and

figure out what the source of that incident was was it internal was it external uh you'll want to figure out does it apply to one bucket or multiple buckets or multiple storage volumes um it could be permission based it could be uh could be a lot of things um if it's malware related you want to quarantine that malware away from whatever production bucket that that file was uploaded to uh so that way it you eliminate that Downstream risk and then if you have governance policies in place again you want to communicate and you want to document everything at the end of the day so um as part of the PO incident actions again restoring data performing

some type of forensics on that data as part of your uh your plan based on this nist framework is crucial and uh just reviewing it doing some type of postmortem um again key takeaways here that I want you all to remember is in the cloud you have a shared responsibility you are responsible for the data for how things are configured you always want to be aware of new threats and you always want to have some type of type of plan in place if the worst were to occur uh and backups at the end of the day so let's get into a little bit about uh actually taking this plan and implementing it uh in a cloud platform

so some of these threats uh that we have so malware uploads um usually you have some type of application or web form or whatever it may be that may be accepting data uh and allowing you your users to upload that data into a storage volume an attacker and it may be an attacker it may just be someone that is unsuspecting that has a infected file is going to upload that file and you may or may not have malware protection at that at that point um if that file gets in it creates that Downstream risk um if it gets in it can't be executed in S3 or in in your storage volume thankfully at least your object storage

volume but that Downstream risk is where things get uh a little ugly um usually when you have some type of malware that's uploaded the biggest challenge is going to be uh identifying what the source was who uploaded it um if you have an application you might be uh scrolling through application logs to figure out who it was from or was it a user was it uh someone that shouldn't be uploading things so on and so forth um the big thing to consider is with this type of threat there's two ways to to think about it one is you can scan files after they've been written to a storage volume and I should note not after

they've ended up on someone's machine but after they've been written to a storage volume or you'll need to decide do you want to scan before they ever get to the storage volume um and that's a personal decision we run into a lot of customers that uh try try it one way and then they'll switch to the other um usually they'll attempt to scan after the file is uploaded to a uh storage volume and they'll find uh it doesn't work for their application it doesn't work for their workflow so they'll switch to having the application send the file to an API endpoint have the scan performed on that endpoint and the result returned and based on that result

they can upload the file uh to uh to your bucket um so just things to consider as far as misconfiguration goes um the biggest thing here is going to be uh not only figuring out who caused the misconfiguration but it's what was actually accessed was anything sensitive accessed um you're going to need to you're going to need to figure that out and fairly quickly um there was one example I can think of uh a company that somehow exposed all of their uh job application data um so resumés and whatever else in a public bucket and they didn't Discover it until a kind researcher uh pointed it out so um you'll always want to make sure that if

something's public and it's due to a misconfiguration who is identifying all of that dat or who is identifying who is accessing all of that data and then as far as ransomware attacks go so uh breaking it down again on the file side if it's an infected file with some ransomware that's one side and then if the storage volume itself the objects within the them are locked up uh it's going to be you know uh it's going to be crucial to recognize what happened um earlier this year there was a threat called uh code finger that was uncovered I think in January uh and this code finger threat applies to uh S3 buckets where if someone were to gain

gain access to your AWS credentials uh they could log in which it's fairly rare but still a threat um they could log in and they can use customer provided keys or apply customer provided keys to all of your objects within your uh bucket so they lock up all of your objects and even worse they'll use life cycle rules to uh Delete the files after let's say seven days and then they'll just leave you a little Ransom note in there that um will give you some instructions to pay them or else uh so if it's a threat like that uh main focus is going to be what what actually happened what user was it what API calls were made to uh

set up those customer provided Keys uh so just things to to keep in mind here all right when it comes to the AWS Services themselves um within AWS out of the couple of hundred different services that they offer um they offer a handful related to logging and uh monitoring as a whole uh the first first is cloud trail so cloud trail allows you to look through all of the different API calls uh for just about any type of activity whether it's object storage related or not and it's a great service to get into after an incident has occurred to see what actually happened at the object level um because a lot of the time it's

kind of broken down cloud trail focuses on the actual objects whereas the uh the actual back and forth the network traffic piece of it is through uh S3 access logs and VPC flow logs so you'll want to know all of these different Services here uh as different types of actions are taken um there's also Cloud watch and guard Duty Cloud watch is great as far as storage goes uh storage metrics go really high level storage metrics um let's say if you have uh a bucket that might have a little bit of data and or on all of your buckets let's say you have monitoring through cloudwatch you could alert for if files are being written to this bucket way too fast if

if there's a sudden increase you could alert for that using cloudwatch but otherwise it's not not that great for for anything um lowlevel and then uh AWS also offers guard Duty as well for not only anomaly detection but also for uh malware scanning so uh they actually recently introduced malware scanning um to their product to their guard Duty product uh last year uh so it allows you natively to scan for for malware now it's a little bit limited but it's a really good uh good native tool to use as well within Azure Azure is uh a little bit less uh it seems like but you have Azure activity logs and resource logs you can use those for Access tracking to

figure out who accessed what um Azure has their own malware solution uh called Defender for storage uh and it also uh from what we saw and used uh it also has some anomaly detection uh features as well functionality as well uh and then in gcp uh gcp has Cloud uh Cloud audit logs for storage access so see who accessed what within your storage volumes uh there's VPC flow logs as well in gcp uh so for Network traffic tracking and then uh Chronicle or formerly titled Chronicle um I guess now it's Google security operations and soon to be whiz probably inside of Google security operations no no uh uh but uh they have Google security operations uh which you can use for log

analysis and tracking and so on and so forth so uh the thing I want youall to keep in mind here is it's really a few different levels one is you want to take into consideration the network level and see what happened as far as uh the back and forth on the network side and all these different platforms provide native tools that allow you to do that uh the second is at the storage volume and object level what actually happened uh uh when you're trying to uh access or or edit uh configuration or or an object on a storage volume uh through the API so there's tools to allow you to do that and then the third piece is um the

actual analysis and alerting all of these different platforms have some type of native anomaly detection um or use whiz you never know so um when it comes to uh attack patterns um when you're looking at these logs there's a few things you'll want to you'll want to know one is um unusual API calls so if is something uh happening within your buckets is someone calling uh the S3 API and trying to list buckets and then access buckets or or uh access files uh or update files update objects or delete objects even worse um you'll always want to have some type of monitoring in place uh for that um along with that large data transfers is another big piece um

we've been building at CSS a new tool like I mentioned earlier that will focus on this data layer and one of the things that we are um uh uh ideating on is this concept of uh impossible travel so if you have uh if you do business let's say mainly in the US AWS regions uh and someone tries to uh write files to a uh a bucket in APAC for example um that's this concept of impossible travel like no one should be using any resources in a region outside of the main ones that you all have uh along with that as far as uh uh an anomalous activity uh the source of whatever is happening so do

you mainly do business in the US do you only have people in the US that are logging into your AWS accounts well if someone logs in from the UK or Germany or wherever else that might be a problem uh unless you have someone there that's traveling there for example um so you'll always want to look out for uh different types of activity whether it's strange API calls uh strange data transfers especially late at night uh as well as where's everybody accessing AWS from where's everybody accessing uh these buckets from so on and so forth um yeah so uh last piece here um let's talk a little bit about uh playbooks as far as incident response goes when it comes

to uh malware uploads it's really really crucial to quarantine any type of object that's found to be infected if you don't that's fine I mean it's up to y'all it's up to you and your organization to figure out the uh the risk tolerance that you'll have but when you quarantine something you're you're eliminating that Downstream risk until you can figure out is this file infected uh what is this file all about um who is it from so on and so forth so the first piece as far as malware upload uh incident response and for any playbooks that y'all have is quarantine the file um second is figure out who it's from so is it from an unknown

public source is it from a customer is it from someone internal so on and so forth and then perform analysis on it so with in uh our uh anti- malware solution at Cloud Storage security we have uh uh static and dynamic analysis functionality available where you can send the file off to our Cloud sandbox and detonate the file and receive a report that says uh here's what the file is trying to do the directories it's trying to access uh what it's trying to execute the IP addresses it's trying to connect to so on and so forth so performing some type of additional analysis on the file is crucial in order to understand is it legitimately uh

infected um something to keep in mind as well every single scanning engine has different virus signatures that are going to be different every single day um and malware is going to evolve every single day so the malware of today the virus signatures of today are going to be drastically different six months from now a year from now so on and so forth so making sure that uh you are performing that additional analysis is crucial because is it could be a false positive um but it could be a false positive today um and you might find that the file is legitimate and send it back to production but if um if you decide to send it back also remember to

continuously rescan files uh in any of your storage volumes so having some type of Baseline created um on a regular frequency like every six months or every quarter whatever it may be is crucial as well as far as misconfiguration goes uh if you have a public bucket First Step usually lock it down uh get the bucket and all of its objects to be uh non-accessible um along with that if it's an incident if it's part of your governance policies you'll want to notify uh your stakeholders internal external your legal team possibly as well um usually as part of the eradication threat or eradication step step I should say in your playbook you'll want to uh identify figure out

what exactly was exposed like we talked about earlier um usually look at access logs to see who accessed what if possible and figure out if this bucket if this storage volume was public was it genuinely accessed or did you get away clean and no one accessed it thankfully hopefully um and then uh the biggest piece that we found with uh uh misconfiguration idence is you're going to want to notify folks uh that their data was publicly accessed as far as ransomware goes um if it's something like a code finger threat where you have uh someone accessing your AWS account locking up your files uh it's going to be crucial to shut that account down if possible uh especially

if they have AWS CLI access and things like that um you you'll want to figure out what was compromised and if you can get it back if possible hopefully you can get it back um and then restore data from backups as part of uh any Playbook that you have uh responding to a ransomware incident especially one that's like a a code finger incident um yeah that's mainly it for today uh thanks everybody for coming by thank you for uh to the organizers as well for putting this on any questions at all for from

anybody yes oh uh the slides so we will put them on our website Cloud storagee security.com uh and they'll be accessible there any other questions

yes yes uh so as far as the guard Duty S3 malware protection tool goes uh they released it I want to say June of last year middle of last year um it's a good tool overall uh it uses the Bit Defender scanning engine um but it has some limitations so you can only scan files I believe up to 5 gigabytes in size um you can and maybe they removed this recently but you can scan I believe up to 10 buckets at a time which I'm not sure why um and the biggest thing that I found interesting at least was when you are um sending your files off for scanning the files are being scanned in AWS accounts

uh that AWS owns so it's not being scanned in your own account but in their accounts which is it's fine but if you have data residency requirements things like that that you need to abide by um you'll want to take that into consideration um it's still in region as far as I'm aware as well but you'll want to take that into consideration additionally they don't have out ofthe boox quarantining um and uh there's just a little bit of extra setup if you want to customize it a little bit with our own solution we have made it a point to complement guard Duty and extend it so um if you need to scan files larger than

5 gigabytes for example we can do that all the way up to 5 terabytes in size um along with that we stand up scanning agents in your own AWS accounts so you manage everything it's in your own AWS account we can't see it we don't have access to the files so on and so forth um but yeah guard duty is a really uh nice native tool if if you just need to scan a few gigs and it's it's um I mean they build it for the masses so if if you have simple scanning needs it's decent overall

yes sure um so code finger was discovered in January of this year uh by a company uh I believe called houseon Labs if I'm not mistaken uh and what they uh released was research on someone obtaining compromised AWS credentials so account credentials which is fairly rare but it's still a threat it still can happen um and somehow bypassing MFA or if someone ignored adding MFA um that's that's specifically what they focused on but someone getting access to those credentials logging in and getting onto an AWS account that has the permissions to uh execute commands uh using AWS CLI and then essentially locking up all of the individual objects in a bucket using uh customer provided keys so they have the

key on their side they lock up all your uh objects in that bucket and then all they do is leave you a little ransomware text file in the bucket you realize you can't access your objects you see this uh text file and uh you got to pay them some money um that was something that that came out can it happen most likely um it could definitely is it happening we don't see it as much you're Mo mostly seeing actual ransomware uh you know in the object itself but it was an interesting um an an interesting way of thinking about it where you have this functionality of customer provided keys so you can manage your own keys but it

can be weaponized um so that's the the the the basics of it um we actually have a paper on if you go to cloud storage security.com we have a paper that covers um uh code finger in depth as well yes

so yes uh that has been something that we've noticed they do a little bit a little bit better on um however I've also noticed that usually it's uh it's kind of one way or the other either they're really really really knowledgeable and they do it the right way or even if they're using iic that still kind of breaks things and and opens things up um the biggest thing that we've noticed though is human error is is Big at the end of the day uh AWS is so large and across all of the hundreds of services a lot of folks are not knowledgeable on every single aspect of a specific service like S3 even even though it's simple even though it's

existed for almost 20 years uh the way you can configure it so on and so forth it's tough and you know it's it it it can get bad sometimes yes just stops the whole attack it it it does indeed not not many do that though for some reason yeah they do do they yeah yeah that's something you can config really easily that's why that Ransom is just noten yeah it's from what we found well I I shouldn't be saying this but I think we were not blocking it from our side so uh from what we found though it you know it's again it's just human error at the end of the day um that it just happens but that's a good

point very good point any other

questions yeah anybody I will take that as a no [Applause] then okay