When cloud infrastructure provisioned with someone in marketing’s credit card... - Aaron Shelmire

hopefully everybody's really energized after lunch not about to fall asleep as you can see I I speak very slowly and very softly so I'll lull you all to sleep and I only have a couple of slides I've got about 78. to make it through and we'll see if we can make it through um so welcome to this session in the shadows of the cloud I'm going to tell a little story to start off with and then after the story which is purely fictional we'll hop into some lessons or things that you can take home and do yourself um so a little bit of a disclaimer The Story You're about to hear is a fictional portrayal well the events may

be inspired by real and reality the details places and names have been changed to protect the innocent if you all remember a couple years ago myself and Christian roylo presented and we were part of an organization called seasides Incorporated we were a victim of the north peche attacks since then our it estate has recovered we've completely rebuilt we've moved off of Windows and off into macintoshes as well as building much of our infrastructure in AWS then we got hit by a trademark lawsuit and I had to change the name to seasides Incorporated or I might have forgot that we called it seasides with the capital c and just made all my slides with seasides the

other way so um it's been pretty busy it's the peak of vacation season and last night I got this slack message from one of our security operations staff um it turns out that our sales operations team has stood up an AWS account that they use to help pitch our vacation packages and it's not part of our AWS organization it's completely separate doesn't use our service control policies at all and yesterday they noticed that it was serving content to sell fake Rolexes that sucks so we need to get access to this AWS account and one of the things that we could do is we could just grab an administrator account and Grant ourselves administrator access via a

role there um that might be a little intense and a little Overkill we could easily disrupt operations in that account or step on evidence that we're trying to gather so instead luckily we have another AWS role already set up for this sort of situation that's mostly built out of read-only access we've got Cloud watch logs and cloudwatch groups access read only we also have Amazon S3 access to check out what is in the S3 buckets we also have our config user guard Duty AWS billing and cloudtrail read-only access so that we can see those events and pull those back as necessary we also have security audit access which allows us to describe and figure out

what's going on with our compute resources as well as databases and uh any other Cloud resources necessary this covers most of it and those are eight managed policies from AWS that we just have to add into this there's a few other things that we want access to to be able to respond to an incident appropriately though so we created a custom inline policy and it has AWS portal view usage this allows us to view the usage of different resources within the organization and I'll add to why that's important in a few more slides we've got ec2 privileges to allow us to create snapshots and create images this lets us image any compromised resources such as that ec2 image ec2 instance that

might be compromised at the moment we also have EBS instances or EBS privileges for everything which allow us to grab disks from our databases as well as our ec2 instances IAM privileges to get and list any credentials roles or users that are around and then KMS privileges to allow us to encrypt the evidence that we're looking at and transfer that out of our account encrypt it into our new account so those that's the additional access we'll put that in a customer inline policy at that point and then we're going to Grant ourselves access to our super secret AWS account of one two three four five six seven eight nine zero one two very similar to the

password on my luggage and we have customer inline public we have a cloud formation template that allows us to just send a link and when the administrator of that AWS account clicks on it it will run and set up our cross account role and Trust relationships that we need to gain access they've done that for us and now we're going to come into the ec2 portal and we know what ec2 instance it is so we're going to go over to that ec2 instance create image modify the permissions to share it with our account of one two three four five six seven eight nine zero and I need to type the one two here left at the moment

first thing that we're doing is a web service so we're going to grab the access logs we want to know what's going on and when this started occurring sadly we only have 11 days of logs to look at but hey let's start doing the world's fastest incident response triage and we're gonna gz cat access log to see if there's any successful web requests to Etsy password maybe Etsy Shadow see what's there maybe maybe they're redirecting the standard into out with a less than two Ampersand to whatever output that they're looking for and maybe there's some but we're not finding that maybe there's some SQL injection with an ampersand one not finding that is there anything with just a colon not

finding anything there we're not finding anything where people are hitting an instance metadata URL which might be indicative of a server-side request forgery where people are using that ec2 instance to access the Privileges and information about accounts and access and roles on the back end we're not seeing anything there so what's going on here maybe we can just browse to the website and see what's going on so we browse to the website oh well let's take a look at some of the logs uh additionally we're looking it looks pretty normal we're getting some registration Pages we're also having some successful logins through the API of verify login we see destinations coming coming back that's all good

let's look at the status codes and uh looks like we've got a lot of 400s a lot of 500s that seems a little odd usually there should be less of a balance of those so yeah let's let's go to this website and see what it is pretty benign pretty basic it looks like just a login page which allows you to log in or register an account there's it looks like it's part of this laravel web framework which is openly available maybe there's something there so let's try to send some malicious or some suspicious or weird things to make this site trigger an error we're seeing a lot of Errors there and this comes up and it turns out that

this site has debug for the application set to True which stores all of the returns all of the environment variables as people hit that site and right below that we can see the AWS key ID and then the AWS secret key this provides somebody enough access to be able to use those to access that user or role role with those keys and it looks like we've got an S3 bucket there taking a guess here but it looks like maybe they had right access or or were able to access that bucket and maybe drop some things in it so let's take a look and we're going to search for this access key ID into IAM console and take

a look at what privileges that user also has access to in this case it looks like this is a pretty good in-line customer managed policy we our S3 bucket is limited to list bucket but then or their access is limited to S3 bucket as well as listing the bucket but then also they can not just get objects but they can delete them and put new objects in that bucket oh I think our story is starting to come together a little bit here and um let's see what's in this S3 bucket all right so this this looks pretty standard we've got a handful of date um directories maybe they're using those to set up a new uh

uh vacation uh campaign and sales campaign for each one of those months and then we've got this dot gcp here and that seems really out of place being that we're in AWS but maybe let's take a look and there's only a Json file inside that folder kind of curious what this is I'm a little scared and that's a service account credential for a gcp account um and and it's for this account for this gcp project of Laguna um and we can see the seaside Sales Group has access to that gcp project um so we're going to come over to our gcp uh console and Laguna is not one of our projects I wonder where this is so

we don't have this I'm curious where we should go to to figure out who's standing or who's using and administrating this gcp project so we're going to send something over to our financial controller and say Hey you know we're looking into some compromised infrastructure in AWS and we found this additional cloud account is anybody expensing things for Google Cloud over each month so we're going to send them this email and while they work on that we still have some more analysis to do and there's some more places for us to look at things within AWS first we're going to go over and look at guard Duty it turns out that they have not turned guard Duty on and that's kind

of expected this is some sales account that somebody has built and stood up separately look at cloudtrail and it looks like they haven't created a cloud trail yet so that's not been turned on either so we're kind of running out of options here luckily uh cloudtrail allows you and stores 90 days of data by default and cloudtrail is on by default it's just that all of the data is going to the API and you have to interact with it through the console now if you want cloud trail data if you turn the trails on and start logging that to a bucket that data is retained indefinitely with Cloud watch that is not on by default but whenever you do turn on

those log groups that stores the data indefinitely as well for infinity and you can configure that data to ship to a bucket as well and then you can work and retain that data and then retire it into S3 Glacier and code storage as necessary with your resources your S3 access logging your load balancers your netflow data that's also not turned on so all of that access and interaction with those load balancers with S3 that's not on by default if you do turn it on also that that stays there for Infinity within your S3 buckets or until you remove it guard duty is also not on by default this is your security detection and protection when you do turn it on it

stays on for 90 days unless you replicate that to an S3 bucket so you've only got 90 days and it's through the console luckily with our cloud trail data we can start to query that through the console so taking a look at our console we can search for it based on access key we know that access key ID from that debug page start looking into it it looks like they're doing some reconnaissance here describing the regions that that has access to and events and such we're also listing access points and describing things around the bucket what kind of encryption is enabled and so forth and you can see the username associated with that access key right

there as well this is really painful but let's open up one of these events and see if there's anything useful hey we've got an IP address here that we can start to look for other activity from sadly the cloudtrail console doesn't let us search based on IP address and we want to do some more advanced searching anyway so we're going to write a little script to hit the cloudtrail API and pull that data back cloudtrails API using the bow23 python package you can just pull back the data from the cloudtrail API and reformat that as though it was logging to an F to a to a cloud trail bucket as necessary S3 bucket as necessary we're going to grab that in

every five minutes just like cloudtrail does and buckets those and then we're going to create the directory path similar to how cloudtrail does that by default with our region year month day Etc and then we're going to Output that data and wrap it in a Json route of records and publish those as gzip Json files what this lets us do is now we're writing all of this out into an S3 bucket and we can pretend that we were storing the past 90 days of data and then we can use aws's cloud trail templates for Athena open up Athena run this cloudtrail uh template data definition and then we can start to query this inside Athena

with a much more powerful query language and with many more Fields like our IP address so we search for that it comes back with the same events we also want to search for get object and put object Events maybe we'll be able to find something here but we get no results because S3 access logging wasn't turned on so we have one more card in our back pocket to pull to figure out what's been going on with this account and how long it's been going on with our AWS portal view usage account you view usage privilege we're able to go into the portal and create a usage report and we can do that for S3 for

that resource and bucket it based on month day or even week or hour and we can see how how much usage we've had for S3 this gets output in the time the time frame that you've set so in this case a month and then we're going to pull that apart and see what our usage was looking like and it looks like something happened in July 2023 there's much many more events one kind of gotcha with view usage is if you try to pull back too many events you'll get this error at the bottom of your S3 report or your usage report that says the report for the period and values you specified was too large and

you've retrieved two tried to retrieve too much data so you need to request a shorter time frame or make those buckets larger maybe looking at a year or a month instead of a day or a week hey it looks like uh Finance got back to us and somebody in marketing Blake Gilroy started expensing a gcp account back in February 22. so we're going to ask them and they've given they've given us access to they're going to give us access to their gcp project now with gcp again we could use a similar roles to the administrator role in AWS maybe just going with our basic roles of owner or editor but we have the same issue that we had with AWS which is

um we could inadvertently trample over resources and evidence so instead we're going to create our own permissions pretty simple the permissions within gcp are slightly different we want the viewer access which is pretty simple that gives us most of our viewage and we can view most of the resources and what's been going on with those we also want private logs viewer and then with gcp you want bigquery user and bigquery job user so that you can start to search that data much more in a much more fine-grained and Powerful language than the logging Explorer console we also want security reviewer privileges so we can see the I am Privileges and then billing manager for us to see how those resources have been

being billed as well we also want uh we might want compute editor privileges so that we could stop an instance or delete one of the Kate's groups that we have and then we want IAM editor privileges to maybe stop using a service account or delete any users that have been added to this gcp project with gcp the logging's much more fine-grained as far as what's on by default and as well as the time periods are much different with gcp both your admin activity and system event audit logs are sent to the underscore required bucket and they exist for 400 Days by default pretty much everything else that's on by default exists for 30 days and whenever

you have to turn things on for gcp those also go on for 30 days so your the other pieces that are on by default are your system event audit logs your policy denied and then your debug logs and system maintenance logs this is usually when gcp has decided that they're going to update your Google compute service on their own and you're like why did my machine go down um the all of the resource logs like your VPC flow logs your GCS usage and your DNS Nat firewall load balancer logs etc those are all off by default gcp also has this weird thing where everything gets sent to these uh privileged buckets rather than a standard bucket and then you're sending

everything in a log sync to anything else that you want to search including bigquery one of the good things to do is to make sure that you're sending them to a bucket as well so that you can retain that data indefinitely so we're going to go into the log Explorer console it's really really it's a really blunt instrument to search and we're going to search for this service account and it looks like they've created a couple additional service accounts in the data below and it looks like there was something in June but most of this data most of this most of these events were in July 2020 or 2022 2020 2023 whatever year it is right now

we also have our IP address and we're going to search it and there's a few less events we don't have that event from back in June so this has given me a good feel that this activity was was going on yesterday and the day before July 19th in the interim our finance folks got back to us and they said there is an Azure subscription that somebody else has spun up Parker Quinn from the business strategy group and it's for the d-sides project the decides project is our project that's uh relying upon using llms to have ai suggest vacations for people instead of people choosing their own vacations because our leadership has decided that AI will know what people want for

vacation better than people know themselves so with Azure it's a lot easier to gain access you just need to invite an external user and um send send them an email add their email address here and provide them the basic role built-in role of reader access through azure Azure slogging is a little in one way simpler in another way much more complex and fine-grained Azure ad logs are on by default if you have a basic Azure subscription you only have 30 days of logs um the more money you pay Microsoft the more the longer they retain your logs those Azure ad logs are where you get your login ability your authentication events Etc um the activity logs are this are for

the Azure subscription itself they tell you what has been going on with that subscription what resources are being created and such those are on by default as well for 90 days and your resource logs for say storage load balancers your ec2 instance and stuff are not on by default now azure's documentation really focuses on taking those different log types and where you can send them not how you get them so if you read the documentation there's a lot of documentation on Microsoft site around taking Azure 80 events subscription logs and resource logs sending them to a to an Azure storage account in Blob storage sending them to an event Hub so you can have that Kafka

stream send off to say your sim or XTR instance or some other thing to perform processing like datadog or what have you or you can send them to a log analytics workspace which is actually backed by Google or Azure storage and you can grab that data out of the log analytics workspace itself or from blog blob storage or you can send it to the Azure Monitor and grab that from their API now what the documentation is not good at telling you about is that you can grab all of your Azure ad events that exist based on your serviced here from the graph API and you can just write a python script and grab those and pull

those back as well you can grab the subscription logs from a rest API and then there are lots of Powershell methods which will and commandlets which will let you pull data from all three logging topics so long as they're enabled and those logs exist in this case this account we don't have any reason to think that it's in scope with the rest of our intrusion and we're just going to search in the log analytics workspace for that IP address and it's never been so nice to see no results from a search so we're going to go back to our web logs and verify that this is the time frame that this IP address has been

active and it looks like yes July 19th and July 20th is all we have at least for our 11 days of web logs that we had so we're feeling pretty good now we're ready to remediate we're going to go into the AWS console and we're going to deactivate the key not completely delete it because we want to leave that alone around so that we can come back and say hey when was this key created maybe what other access did it have access to and we don't lose those permissions policies as well and then in gcp we're going to come in and we are going to disable that service account not deleted as well so that we can make sure that we have the

other um we we have that additional metadata around what permissions that had at that time so that's the end of my fastest Cloud incident response story ever and it's while this is fictional um it's kind of a enamalgamation of many of the stories that the incident response teams at my organization at Palo Alto networks have been um encountering we see a lot of these instances where really large organizations end up having a cloud instance stood up a cloud account stood up or a gcp project or Azure subscription stood up by somebody outside of the I.T and security organizations it's not part of the organizational policies and sometimes it's just somebody's personal account and they were getting frustrated with the service

control policies that were in place and they just stand up and start using their own account instead and they'll start to expense those back to the finance group there's almost always a tie back to finance and expenses the other thing that we see is since these organizations aren't engineering or I.T organizations um frequently those web services will have been stood up in debug mode and there'll be leaking credentials or there will be some other misconfiguration that allows you to gain access to things the server side request forgery method is uh I feel like heavily played but I see the debug and the keys or service account credentials from your environment much more frequently than that to tell you the truth

you know in a lot of organization well I feel that in in my experience there's if you're in a technical organization from the startups that I've worked in usually it's about when you hit the point of a hundred people there's probably an account that somebody has that's a personal account that they're using for a business critical service so if you're in like a more technology startup I guess like an engineering company or a robotics company if you're in a bigger organization it takes a little bit longer but if you have ten thousand twenty thousand employees you probably have a few of these accounts that you don't know about and it's kind of scary so what can you and what can we all do

about this so one of the first things and I I didn't really hit about upon this explicitly but one of the first things to do is agree upon what your incident response strategy is um and I think this is a overlooked area for a lot of organizations and it comes down to what how are you going to approach finding compromised assets are you going to remediate them as you go and continue to play whack-a-mole and just deal with one at a time the entire time or another strategy that I've seen a lot of organizations take and and use is this concept of a rolling remediation and eviction so they're going to scope and they're going to continue scoping but there's a

lot of pressure for them to start remediating and stop the bleeding so they're going to investigate it maybe for a week maybe for a month and then every month they're going to roll through and remediate what they know about the approach that I've always preferred is to do a wholesale analysis and eviction at once where you continue doing scoping until you've figured out the entire uh extent of compromise in as much data as you have available to yourself now the downside of taking that approach is if you're dealing with a ransomware or destruction scenario you may only have one or two or three days to start remediating and containing that damage from that actor doing a much more Deep dive into your

forensic images of those hosts and such might take 48 hours or more for each one of those hosts that you find compromised so you've got to play the balance of those two items the other thing that I I wanted to hit upon was you know what do you need and what should we all be preparing for as Security Professionals within our organizations so one of the things that I recommend is that everybody has a has two accounts at least two accounts per each cloud provider being one a security account and you use that Security account in order to gain access to additional compromised accounts and if you are using cloud services within your organization already then you would use that as the

hop-off point into your organization's allowed cloud services and accounts the other thing and from my experience I manage all of these for unit 42 and um our bills are pretty low for all of these for for Azure for gcp for AWS it's a little bit more expensive that's kind of like where we have a lot of data being stored and it's still just a few thousand per month most of them it's under a hundred dollars per month for our sandbox accounts where we let people play and stand up flaws.cloud or vulnerable resources or try to gather evidence from those see what that would be like those accounts are usually around 10 or 20 a month it's extremely

cheap sometimes there will be a little bit more cost associated with those where we have to pull down say a a privileged Ami machine image that costs 400 a month or something and hopefully you'll be able to build that back to whatever department is using that Ami image the other things that you need for AWS have those cloud formation templates ready to go both for administrator access and then for your investigative role the investigative role having mostly read-only access and then some additional privileges that you need such as the view billing resources um and then also having those Python scripts ready to go for cloudtrail and Cloud watch cloudwatch you can also grab from the API even if it's not logging to

a bucket and you can replay those logs and then drop those into a bucket as necessary to perform long-term analysis and get used to using the AWS command line interface in Botto 3 which is the python module for AWS for gcp you want some terraform with the permissions that you want probably admin privileges editor privileges to the project as well as your more fine-grained and mostly read-only privileges gcp you want the gcloud command line interface as well as their Google python packages the Google python packages do take I'd recommend definitely spending some time playing with them Google really likes to use iterators versus just passing back the rest response and getting used to iterating through that

data and working through that data in that way is uh it takes a little bit of time with Azure there are a lot you'll want terraform for the access and then there are lots of Powershell scripts available to perform log acquisition and pull that back they're pretty comprehensive as well I'm personally fairly allergic to Powershell and the Azure command line interface is nice and there's a lot of azure python modules as well and Mystic Pi has a lot of packages ready to go to perform analysis based upon the log analytics data that that can land as well as Sentinel data that are extremely helpful oh okay the last couple points so usually whenever you buy security

tooling and like your sim or your xdr tools those are made and and the configurations for those for and Integrations for your cloud systems are made to set up say an sqs topic and start publishing events from create object events in S3 or a pub sub that is looking at create events inside GCS and then in Azure you'll have that Kafka event Hub set up and streaming those same events as well from the log topics over into um uh over into your sim but those are all made to go from hey we bought the Sim and we're looking at things from here forward or we've integrated the Sim to this um to This Cloud account and we're

looking at things from today forward by using those scripts from cloudtrail cloud watch or if you already have data going to S3 buckets and you discover a account and you need to do forensic uh response looking at data from the past if you set up an sqs topic and set it for create object events um when you do a copy of that historical data the Sim will still ingest it however a lot of the machine learning and more analytic-based detections usually those are based upon ingest time not event time but you'll be able to ingest all of those historical events and then you just need to start performing analysis looking at the event time of things rather than you're in

just time the last piece is decide upon your primary Cloud whether that's gcp Azure or AWS and then within that primary Cloud set up Imaging hosts that allow you to perform light forensics for Linux and windows and such there's a lot of triage scripts use OS query what have you and then also you're more in-depth forensic Imaging using DD to grab that image and then you know loading that into Axiom or x-ways or what have you the next piece is you want to have templates for all of your core data sources firewalls the different Cloud providers and different kind of security tools that you may have create those templates so they're ready to go within whatever your primary cloud

is and whatever that data platform is that you're going to use whether that's Athena kql for Azure or bigquery and gcp um and then the last piece is partner with your Finance team like I said earlier um usually whenever people stand up these uh Cloud accounts they will want to expense those back even if it's 50 a month 100 a month they're going to start expensing those back and by looking at what things are expense through your organization you'll be able to discover that somebody has a recurrent bill for twenty dollars a month to AWS so that's uh the extent of this session I wanted to say thank you these days mostly I'm uh I'm an engineering manager in front of

many teams I don't really do a lot of the work Hands-On myself sometimes I get to help people out but I need to extend thanks and kudos to my team they're the ones that have mostly discovered or figured out a lot of these things and I'm learning from them Additionally the Consulting team has a lot of engagements we're usually dealing with two or three Cloud engagements every week and I get to hop in and be the rubber duck for them to run Concepts passed and sometimes they figure it out by themselves sometimes I get to help out a little bit and of course thanks to the B-side staff and everybody that came and attended this session

thanks [Applause] um