Cloud IR: A Rapid Guide For AWS, Azure & GCP

Thank you very much. Good afternoon. I hope you still have energy to listen to this topic. Uh so short introduction. I will talk briefly about threat landscape a report that IBMX Force publishes actually published last year about the problem and something that I want to talk with you today and to share my insights. Uh and then I have two deliveries. One is kind of uh cheat sheet sound style which basically it's designed for for forensics analyst incident responsor to use in case that you have kind of cloud uh incident response and in the end is a tool which mostly I I develop with AI. So please don't judge my programming skills which basically helps a bit just to uh have

some uh readiness and to be prepared for incident when it comes to cloud and the first version covers only AWS but no spoilers I will briefly talk about it in the end. Uh who am I? Basically I work full-time for IBMX force. Uh you I'm incident response consulant. I'm so I'm part of uh X force incident response and I lead the cloud incident response for IMIA and basically X force is a team of hackers within IBM. Uh basically we support the our clients worldwide when it comes to different services. My team usually works most of the cases uh on reactive side. It's purely incident response uh retainer but in the same time we have as a proactive services

which basically we help the clients to be prepared for uh for the incidents. Uh I worked last seven years detection engineer sock manager a bit of everything. Originally I'm from Kosovo but I moved recently in in Germany. So I'm looking forward to to to build kind of community here as well. So really it's a pleasure and looking forward in networking to meet all of you and as well feel free to contact me after after this. Uh so this is a report which I don't want to go in more much details because because this is public but I want to basically just break it a bit in the pieces and to focus on the the piece

that I think is the most important for today presentation and the main giveaway but before I go here I think the the main part is that uh there is huge gap in cloud especially cloud forensics not just like on clients side but as well in community. So my first question maybe I will start who who has ever worked in cloud forensics or cloud incident response I expect few okay so it's good number but still it's kind of huge gap and I think this this this the venue is very concentrated people so imagine now worldwide uh so first I will go there are some basically key findings from this report which is as I said this online uh first starting

with fishing and exploitation of vulnerabilities as an entry point. Still you have basically the legitimate credential used for initial access is very common that you have info steelers. you go in dark web the price per credential is dropping which means that is very very common and it's a trend basically that threat actor also use cloud for for for malicious activities in this case which basically uh this reports give you more details but the main focus that I want to do to to focus today is the on recommendations so basically there are some different recommendations starting from preparation testing red teaming blue teaming security testing uh building the posture having some testing mechanisms some some cloud providers who basically

provide you stronger identity access AI AI AI it's basically the most common topic those days but something that I want to highlight is basically how to strength incident response capabilities because I think it's something that is very common and I was just speaking with one of the organizers users here before I I I came here. You have cases basically you have onrem you have ransomware attack for for for example and what what is the final output or outcome of this you pay or you not pay the ransomware but we have cases and basically the third actor exploits one of the initial access points they manage to get high identity or admin tenant whatever and they spin Bitcoin minings

and imagine if tomorrow morning you wake up and the bill is I don't know 100k because of high GPU and stuff. So good luck negotiating that with Amazon or GCP or any other cloud providers. So it's very common and and some sometimes uh we forget about these capabilities and the main delivery for those for this presentation is basically to focus on this. So how to be prepared in this case. So before going there I want to talk a bit as well not just because of logs forensics a colleague colleague of mine he shared like very specific details on rare components and rare uh operating system I don't think that cloud is rare so it's very common but

the point is that sometimes it's a bit insecure by default why because cloud providers they usually choose uh basically default configuration they want to have accessibility to be more faster to adop faster to increase faster and not by default the uh security is a priority in this case. So I think that when it comes to usability versus security, it's always usability because they want to have fast rapid grow delete everything. For example, if you take lambda is extremely convenient, but in reality the logs if the machine or the lambda is gone, you will not you will not have basically a classic disk image to go there doing the do the forensics and it's default is open is accessible. For

example, you create the machine SSH will be open. You have public endpoints. You have a lot of things which I kind of understand because the main uh goal is I think is a profit. So of course sometimes we forget. So it's kind of insecure by default but in the same time the shared responsibility confusion not all the clients understand that basically if you spin up a VM in in cloud it's it's going to be basically responsibility of Amazon. But of course it's like a model. There are predefined rules there that you have and basically that's those are the components that which I think that it makes cloud uh insecure by default. But now when it

comes to basically doing incident response in cloud there are even some other components which I don't know what I'm touching but [snorts] here is basically the logging problem. Not everything is on by default. For example, for if you create a tenant, you didn't touch anything, you might have only cloud trails enabled, which is going to be for a default retention. So when the days pass, everything will be gone and uh that's lack of logging is equals with lack of evidences and then good luck finding the road calls in this case. uh so it's not just about overall logging but as well for visibility in this case is vertical or horizontal how many accounts do you

have for example you you have a very secretive uh account who is for proud but in the same case you have another account who is basically used for for development purposes and it that when it comes to horizontal uh uh uh visibility so do you have visibility in all accounts and as well this is uh the hierarchy of needs for incident response. Uh it's publicly available in GitHub. Uh swam man I don't I don't recall his uh full name but it's very cool uh concept which means that can you do remediation in real time only if you have those kind of uh capabilities in this case. So basically if you want to do some fancy deception you want to

basically uh act in the same tempo as a threat actor then you you have to be very high in pyramid for example if if you don't have the inventory you don't have a teleamentary you don't have a detection it's very hard to get anything and sometimes in cloud cases we don't even have this inventory in this case which I think is very easy but it's kind of confusion at least from my from my uh perspective in this case. So the point is for example you want to check uh or you want to hunt or track threat actors that are basically going around then you need some good deception but of course you need basically all the logs that are

needed for this case and you cannot jump directly on basic ner mediation without having the basics. So this is very cool concept when it comes to first if you are working as a CISO or security manager or security detection basically this helps you to set some expectations but as well if you're working as a consulant and you are a client is approaching you but they don't have anything everything is just the defa by default logs and of course you cannot set expectations because you are missing the basics. So really like the basic of a pyramid or low level of pyramid is very needed and you cannot go up if you don't have the basics. I think this is

more focused in like overall IR but I expect this the same thing uh basically is needed for for cloud cases as well. So those are the problems and something that is working at least in our team is a cloud IR cheat sheet which I was inspired by son's 509 cloud incident response training which I did and basically the cheat sheet is more abstract so it's not that maybe you can disagree with how I split the levels for example starting with management plane everything that is on API level which basically covers cloud trailing in this case in AWS. Then you have network component compute application data and in the end those cloud security services. So the columns you have the

category in this case uh you have the logs which are available uh at least for the time being maybe something was changed and in the end you have some scenarios or related TTPs which basically are applicable for for the incident that you are working on. So for example uh management play logs are on by default usually and if the client is trying to to know for example what kind of TTPs are touched for example creation of uh shadow users or disable of logs or something that is on API level and of course you might have the answer but the retention is only 90 days so if you don't have basically everything shipped on S3 bucket or in

external uh sim then of course this is very very hard to get the answer because you don't have the visibility or for example the there is a requirement that you want to know what kind of data was excfiltrated from the bucket S3 bucket in this case and on the API level uh on cloud rails you will have everything that it was touched via API so via console or basically you are using a common line but if you are going through the browser HTTP access you need uh basically S3 server access data which basically is not on by default is very noisy but is very helpful for for sensitive components and then you have other components for example you have VPC logs

which on prem and in traditional machines it was very complex very very uh expensive solution I would say network tabs mirroring or just net flows here is very Easy. So I'm not seeing too too many cases when the nets are are on by default. And as well for example if you go to let's say you have an EC2 machine uh you have agent you're using I don't know Splunk SIM or EWS config but imagine that you have a load balancer in front of those machines and if you are basically not having load balancer logs you are investigating only the EC2 instance itself then all you will see is the traffic between you and and the load

balancer so it's going to be huge huge gap when it comes to investigation. So the same thing is for Microsoft as well. Uh I don't know it's it's kind of different naming convention. I think it makes the cloud more complex especially to the people that are new in the in the in the industry I would say or new in the cloud. But kind of the same thing different naming convention but in logging I think it changed a lot especially when it comes to IM in this case and here for example the same things. So you have management plate but you have different cloud. So in AWS you have only the cloud rails but here you

have for example ant ID which is of course makes a bit more complex you have uh net flows you have asure monitor agent application data and security services by the way if you want to know more about tps that are used you have as threat matrix and you have the same for EWS as well which can be a good reference so here the same thing you have a machine and the threat actor is very common that they use a run command basically to have kind of back door and you need some very special logs for that. So it's it's not going to be on by default. So you have to go in in in details for that and cloud

Google cloud platform the same thing it gets a bit more complex because uh Google cloud will treat this as a service itself. So it's basically uh something that it matters how much are you paying what kind of service level do you have I haven't included anything yet here but here is the same thing for example VPCs compute application level etc etc which basically you need to go there learn a bit how it works and then just match with TTPs that are available in miter or in this case and see for example what kind of uh answers do you want to to to provide to the client or basically your your whoever the project that you're working

uh when it comes to this. So that's it. Uh that's the cloud cheat sheet which I think is very nice to have printed in front of you when it comes to triage calls for example scoping and you want to set up the expectations for internal team or external client. What do you want to do? I want to have basically to know what kind of a situation happened in my in my uh company. Do you have server access? No, I don't have basically the answer is we cannot move forward because that's the limitation of cloud and I think it's kind of impossible to get the classic uh disk image collection to go to the data center because it's kind of impossible.

And then I wanted to basically do something for this to not be fully uh manual because you need to go one by one. And like few months ago I was thinking okay maybe I need to learn a bit coding and basically explore this options to have something more automated way and I was inspired by EWCS which is NATO flight for early warning. They check the the skies, the clouds and stuff. And I was thinking, okay, can I do something similar which basically first saw with AWS can you can check first your your level of of logging? Do you have the minimum level and basically if you can share it basically with your client in case that

you are helping to respond and EWS ACS checks for visibility and log coverage and basically it reports the gaps that might uh make more difficult for you to basically do IR and to get the answers that you need. output is console in real time fancy one with different colors and we have the CSV output basically which you can check so those are the services that I check cloud trail AWS config VPCs and it checks for the region uh it's not yet available in GitHub but I will publish it so it will be open source and what is the output is like this so basically we test it very funny because uh me and a colleague of mine and we

manage we have kind of a playground for AWS and account and I was expecting that we both work in cloud security we will have everything enabled but see the result only cloud rails nothing else because why we didn't have incident yet and you first need to be impacted you need a hard hit from the thractor then you will start to move so basically it's a Python script uh use the SDK function and here you will see you will start basically you have some pass if the login is set uh information is more uh I wouldn't say lowle interest or or low importance but it's more you need your need depends on your organization context and the basically

the fail is for the critical ones uh for example the logs cloud trail management enabled it means that we have fall trails. It's a good news but it's only enabled. So what do we need to do is we need to push those logs in sim or s3 bucket or somewhere and we can basically uh do the investigation and then you have uh basically cloud trail data you have cloud trail insights you have VPCs you have elastic search you have a lot of things which basically you need a bit to understand the context because you might need you might need it but the high ones is something that I think is extremely important to have when it comes to

incident response in cloud in this case. So this is a tool. Uh basically my uh biggest highlight for this presentation will be take a look as I said is very beta. I'm just playing around with it. If you want to contribute feel free but the point is just to share something which I'm seeing not only on clients not only in the community but something that is a is a gap and you need to have it before the incident because after the incident I think it could be very late. some resources as I said my GitHub cloud threat report is on IBM site you have MITER cloud matrix which is very important s training I think it's very

good one you have thread cataloges by the way the sounds 549 GitHub is public so you can access it there are some cool scripts and basically that's it I'm open for question discussion whatever you you you wish to to basically I'm happy here I'm I'm here. I'm happy to answer. [applause]

>> Yeah. Thank you, Evelyn, for this interesting talk. Uh, do we have any questions? >> Yep. >> Thank you. Um I would be interesting how many or how much ransomware or extortion based attacks do you see against cloud tenants that you manage or help good honestly I think the biggest problem is is mining more in in cloud uh because it's easy you have high GPUs and it's very easy for the actor to get the money uh It's common but not as common as as a normal uh or on-prem solutions but it's a threat that is present I think that it will be for a long time but luckily I think that cloud is more prepared for runover cases because you

have some controls you have some immutable >> controls you can set for bucket which is nice but in the end I think the problem is uh we have some culture we have some education basically how you can speak with infrastructure team and if you mention domain controller everything is super sensitive is a focus but in cloud it's all about IM in this case and I don't think that all [snorts] administrators of cloud understand the importance of of IM because it's it's a domain controller it's uh firewall it's everything and sometimes and that's as long as this culture will be in our teams I think that not just a ransomware but everything will be around for for a

very long

Hey uh thank you Arbin for the presentation. Uh I have a quick question regarding what's the biggest challenge you as a cloud incident responder face when you know when you have to conduct an incident response like you say ah crap here again logs. Okay, [laughter] it's really like just the logs because okay you you have a 3 hours call you see the client it's clear what happened it's but then nothing is is saved local uh externally and it's very easy for the threat actor basically to go and if you are saying everything locally in the same account to delete everything so I would say logs are are the biggest problem which I hope this would help a

bit soon right thank you

That's it. >> Okay. [laughter] Um, did you have incidents where logs were not there? For example, um, just like Microsoft didn't lock um, AI requests or AI accesses or anything like that. Did you have any anything like that where you just didn't have a chance to get the locks because they were simply not there? It's very very common. So it's it's very common especially in the companies that just moved to cloud. I think it's very common that logs are not there. And as I said in the beginning, good luck with it because you won't have a second chance. Basically the the previous presentation was for forensics. You got the steps. You have different artifacts that basically can

tell you if if there is no logs might be MFTt in Windows for example there might be some artifacts which are not by default designed for forensics but they can give you some answers and and basically in in cloud sadly it's very common that you will not you will not have that especially in smaller small to medium organizations because of course there are some major companies and as I said there are a lot of vector for example, publishing API keys in GitHub, leak credentials or just misconfiguration which basically will impact it. Any further questions? No. Then please another applause. Thank you very much. [applause]