Learning from AWS Customer Security Breaches

uh but let's introduce our next speaker uh ramy mcmccarthy he's going to be speaking on learning from aws customer security breaches uh in light of the increasing cloud adoption there have been a broad coverage in on a compromise of customer environments in the cloud anything from the simplest open s3 buckets to deeper and much broader tactics that are used by attackers and researchers to compromise cloud environments so this talk will focus on aws and discuss over a different dozen different public breaches so let's welcome ramy mccarthy thank you hi folks let me get my slides up on screen

awesome so thank you for the introduction thanks everyone for tuning in to this talk and of course thank you to the organizers of b-sides connecticut for putting this all together and for having me this talk is learning it from aws customer security incidents and we'll dive right in uh first before we get into the core content some quick background on me my name is ramy mccarthy and i'm a former security consultant turned product security engineer i'm aws certified in security and hold the ccsk from the cloud security alliance i'm also the creator of sad cloud which is an open source tool that lets you create configurably insecure aws environments via terraform and i'm contributor to scout suite which is an

open source multi-cloud auditing tool so let's talk aws customer security breaches here's the plan for the next 28 or so minutes first we'll do a quick background on the cloud and aws security to baseline everything then i'll introduce the main topic of aws security instance i'll detour and tip my hat to some prior art on this topic and then dive in and we'll go through the case studies that make up the meat of this talk finally we'll pull out some trends from these cases and i'll give recommendations to address what we find are the most common causes of real world aws breaches so we need to start by we need to start by establishing a

working definition of the cloud in essence the cloud is just software and services that are running on the internet as opposed to locally on your computer or private network that's the entire difference in this talk we'll be focusing on aws which is the largest public cloud provider by market share about you know a third or more of all companies working in the cloud or running on aws so while we're looking at aws here and the cases we'll be looking at are all aws specific i do expect that many of the vulnerability trends and recommendations will be generalizable out to the broader cloud computing landscape it's important to note this talk isn't intended to criticize aws or compare aws its general

security posture to any other cloud provider um but as aws is the dominant platform currently it has the most public examples we can pull from so uh before we get into examples this is the shared responsibility model and every talk about aws should start with a discussion of the shared responsibility model it comes directly from aws and its purpose is to break down in detail the balance of responsibility and securing your cloud account between you and aws as you can see based on this visual aws is responsible for securing underlying resources and services well you the customer need to own security for your workloads as well as configuration that means that when you operate in aws

you don't have to worry about you know securing data centers for services like s3 you don't need to be concerned directly with api security or the security of the underlying infrastructure however you're still responsible for configuration and that includes authentication authorization firewalls and a bunch of other things the focus of this talk is going to fall entirely on the customer side of this model so before we talk about specific cases there are a few points i want to hammer on first uh the cases we discussed do not represent any breach or exploitation of aws itself i'm not gonna be talking about any you know zero days in aws instead they're instances of specific incidents with individual customers

and this is generally going to be due to their usage of aws or how they configure the services they're using and in some cases due to unrelated security posture of these companies at the time i also want to make sure we enter this conversation with empathy incidents happen and there's no such thing as perfect security my goal is to keep this talk in line with this you know principal blameless postmortems on this slide is a discussion of the concept pulled from google's site reliable and site reliability engineering book blameless postmortems focus on the root cause of an issue and try to remove any focus on the you know people involved this is based on the idea that incidents

are not the result of individuals but are broader failures of process technology etc and these instances should be owned by the whole organization and looked at holistically with that said we're going to talk about some public ews security incidents and uh you know i appreciate the those of these organizations that released postmortems publicly that sort of transparency is super valuable to us all and so i really want to highlight that as a positive as opposed to criticize any of the organizations who you know have unfortunately suffered a breach first i want to touch on the most common and most publicized incidents in aws as a broad class which is the accidental exposure of sensitive data by

misconfigured services which allow unintentional public access this is most broadly discussed through the numerous you know broadly covered examples of s3 buckets being open and breached but can also include things like misconfigured elastic search clusters i'm not going to explore these in depth in this talk just because i think it's been covered at length the links on the slides there do give you some kind of compilations of these sorts of interests if that's something that piques your interest and you're looking to learn more i'll also share these slides at the end of the talk so all these links will be accessible the other type of instant i want to highlight without going into specific cases really

is ransomware ransomware is a type of malware that steals data encrypts it or both and then threatens to release or retain that data unless the attacker is paid a ransom ransomware is most commonly discussed in traditional environments recently it was in the news due to hitting a lot of hospital networks but ransomware can also target cloud environments and is growing more common there there's no good single case to discuss because this is a more recent trend but i won't give the high level ransomware can target either aws services or user managed resources running in aws so you know your redshift or maybe rds is an aws service that could be targeted or if you're running my sql server

on an ec2 instance in the cloud that also could be a target generally the root cause here i've seen is internet exposure combined with a weak password uh and then of course a brute force attack against that password grants the attacker access and they can start you know ransoming the database i've linked to some forum threads with users reporting this in their environment just to show you that this is an active threat and i've also heard from some peers who've seen this happen at their companies but those incidents aren't public yet it's definitely something to be aware of on the horizon but again there just aren't some good cases to point to so i call it out here

as promised i want to provide some pointers to other discussions on this topic and you know tip my hat to previous talks and perspectives on aws customer security incidents first off spencer gave a forward-looking talk at defcon's cloud village this year where he discusses kind of what the future of ransomware in the cloud will look like with the mechanisms for ransom in the cloud are if my earlier mention of ransomware if this is a talk that interests you spencer's talk's a good place to start there were also two talks on cloud security incidents at the sans cloud security summit this year first from dave shackleford and then from firearm india um and james condon gave this talk at the

inaugural forward cloud set conference this year where he highlighted some cloud threat actors finally f5 included a bunch of instances part of their is the cloud safe series uh and again all of these uh all the instances in all of these talks are gonna be present in this presentation but they're a place to go to learn more now it's time to move on to sort of the main event here which is going to be a survey of aws customer security instance for each one i'll give you a pointer for where to learn more but i'm going to cover just the initial vector of compromise any escalation or pivoting mechanisms that were used and the resultant impact as a result of

the breach or incident so the first one we have to talk about is probably the example most people will be likely to be familiar with which is last year's capital one breach in august of last year a former aws employee exploited a misconfigured instance of mod security which is an open source web application firewall and this was misconfigured in a way that let them perform an ssrf attack on the metadata service and to break that down a little more a server side request forgery attack is where the attacker manages to get a victim server to make responses on their behalf and return the response they can control the victim server to that extent in aws this was

particularly high impact as an ssrf vulnerability could retrieve the credentials that were connected to an instance and then gave the attacker aws permissions in this case the role stored in the metadata service was over permissions this is a web application firewall that had some aws permissions but for whatever reason in this environment those permissions included undue access to sensitive data from capital one's s3 buckets the data included over 100 million credit card applications that were stored in s3 with all sorts of uh you know sensitive data thankfully the attacker was arrested and had never spread the data so this didn't make it into broader distribution but um is still a really notable incident and got a ton of press coverage at the

time this next case is pretty much a worst case scenario and is often bandied about as a horror story effectively cloud spaces sorry code spaces was a cloud-based code hosting collaboration company and they were breached all the way back in 2014. attackers somehow gained access to aws console credentials for code spaces through an unknown vector a lot of peel suspects phishing but we don't know that for sure they also simultaneously were performing a denial of service attack that was you know a distraction for this breach the attackers created backup logins additional accounts and keys and when codespaces attempted to regain control of the account the attackers began wiping data s3 buckets ec2 instances snapshots in the end codespaces was

actually completely unable to recover and went out of business as a result which is why this is such a notable incident it's it's shown that the consequences of a breach can you know ruin your company effectively next up is the dmc hacked by the greu during the 2016 election i pulled these details actually from the dnc's lawsuit which is linked at the bottom there the initial vector here wasn't disclosed but the result was a compromise of test instances of tableau and vertica which were being used for some data analysis the attackers exfiltrated the data by copying ec2 snapshots to accounts they controlled and they walked away with not sensitive data but actually the queries themselves

which were considered sensitive by the dnc as they gave away some strategic you know signals and this was only with the breach of a test cluster so again something to think about there a lot of people secure their test and production environments differently this goes to show that you need to be really conscious of where your sensitive data lies what you care about and how you protect it the data dog case is in my opinion a much more standard example of an aws security breach datadog had an incident in 2016 where the aws access key and an ssh private key used for their ci cd pipeline think something like jenkins were leaked however they don't know exactly how these were

leaked or at least it wasn't disclosed but generally this is via accidental commits to public give repositories which we'll see and discuss in some of the other cases the attacker in this case was able to use the keys to access a small subset of ec2 instances and s3 buckets and datadog's account and then tried to use that access to pivot with customer credentials because datadog is sas software that was unsuccessful thankfully and the attack took place over just like seven or so hours um and was identified in quarantine shortly after uber had a 2016 aws incident which wasn't disclosed until 2017. here again we're seeing aws credentials accessed via a git repository in this case it was actually a private

github repo but it was lacking multi-factor authentication at the time which was a contributing factor attackers were able to access personally identifiable information on 57 million users as well as names and driver license numbers for 600 000 drivers the case had an interesting twist actually and i recommend you kind of read up on this because uber paid the attackers a hundred thousand dollars as a bug bounty um to resolve the breach uh and that was paid through hacker one but the incident continues to make waves uber's former chief security officer was just charged back in august i believe uh by the department of justice for allegedly covering up this breach and there's definitely a lot of interesting

um discussions around that and important to think about as a security practitioner one login's 2017 instant is another case of an unknown origin and somehow aws keys were compromised in this case the attackers created ec2 instances within the environment and used those to perform further reconnaissance they moved on to access database tables that had sensitive data but in fact some of that data was encrypted unfortunately despite the use of encryption here when login wasn't able to confirm whether or not the attackers had also accessed the encryption keys and as a result they had to treat the data as compromised as a whole again as we're talking through these instances if you're administering an awareness environment i definitely

in the back your mind think about how and where you would be detecting these incidents uh what you would do to resolve them and i'll go into some recommendations at the end politifact had a very different type of instant in 2017 and there's minimal information on it because it was fairly small but effectively a twitter user researcher noticed coin hive running on politifact's website um coinhive is a monero crypto mining script so basically it uses some or all of the browser resources of a person visiting the site to try and mine cryptocurrency politifacts executive director did attribute this to a misconfigured cloud computing server um but there's no further information as far as i'm aware

but the la times had a 2018 incident that was very similar with coin high crypto mining in this case it was directly attributed to an s3 bucket with global write access the attackers were able to add that crypto mining code directly to the s3 bucket because of that misconfiguration and so it's likely that politifact had similar causes tesla in 2018 also had a incident in their case a misconfigured kubernetes console lacked password protection which attackers used to spin up resources and mind cryptocurrency and so this was similar to crypto mining attacks in this case the attackers instead of using kind of browser resources actually spun up inside tesla's aws account resources they used to mine

cryptocurrency imperva also disclosed to breach in 2018 and in their case they had externally exposed a compute instance and it was globally accessible and they said it contained an aws api key and it's not clear whether this is you know a key stored directly in the instance or if this is another incident with metadata access potentially like capital one that access that the attackers gained was used to steal a database snapshot that had been created during testing and kind of forgotten about so again think about a lot of things that went wrong here and imperva's blog talks about their lessons learned as well as do many of these postmortems which makes them so valuable in a fairly novel example a recent plea

agreement uh highlighted a 2018 cisco security incident which in that case a former employee um who hadn't been working for them for about five months or so deleted a bunch of ec2 instances in their account this is interesting because there's some speculation that this was just a complete accident that the engineer was trying to you know destroy some infrastructure in either their current employer or personal using infrastructure as code something like terraform or cloud formation this hasn't been confirmed that this was an accident and that was the case but it does show that malice is not always a component of an incident jw player is another victim of cryptojacking this time in 2019 in their

case weave scope was publicly exposed and that's a container monitoring tool that has rce by design the tool has code execution functionality so the attackers could just directly use this public tool uh to spin up resources and mine cryptocurrency malindo air is another new vector we haven't seen in these previous cases former employees for a contractor of theirs used their authorized access and expanded it to pull down 35 million customer records they were not approved to access and twilio like the la times uh this time in 2020 had a s3 bucket with global write access in this case it was opportunistically exploited so they weren't specifically targeted here and an sdk they provide was backdoored

with mage cart which tries to do credit card skimming generally in this case the instant was quickly identified and resolved um and there was really no impact because the sdk that was backdoored had nothing to do with credit card processing so quick sidebar on mage cart and s3 global right i'm going through these cases this is non-scientific and is going to be much more qualitative than quantitative throughout this talk but i do want to point out that there are far more instances of s3 global right being an issue for organizations than i'm talking about here risk iq in 2019 uh discovered that you know magekart was hitting over 17 000 uh domains or organizations via s3

bucket global right so this is a well-known and broad issue even though i'm only going to call out a couple specific examples all right just a few more here expel is a sock as a service provider so they provide managed security services and they've blogged about a couple of the instances they respond to and these blog posts are really excellent excellent and i recommend tracking them down um if you're interested uh in the first of these a root im user access key was somehow compromised and attackers generate ssh keys for a bunch of vc2 instances and then use those for crypto cryptojacking again we see you know using computing resources to mine cryptocurrencies one of the

common results of these breaches as it's an easy way for attackers to turn their access into cash in the second of these blogs eight im access keys were compromised those keys were used to back door security groups to allow the attacker's internet access into the environment to ec2 instances the final impact we don't actually know because once attackers had command line access to ec2 instances there was not sufficient monitoring or visibility in place really to determine what they did with that access um so again a lesson there about monitoring and where you need it in cloud environment the team tnt worm is an escalation to these cryptojacking attacks because it automatically scans and exploits misconfigured docker and kubernetes

platforms and has recently added features recently as in this year to steal aws credentials from the directory in which they're stored in plain text so far it's only been observed to do cryptojacking for monero and to pass those credentials back to the control server um so it hasn't actually used these credentials yet but that's only likely to change as the uh you know this and similar worms mature crypto mining ami uh this is another vector again for cryptocurrency mining uh abusing cloud incidents in this case a windows 2008 server ami amazon machine image had been around for five years was identified that contained a monero miner so amis are pretty much templates for spinning up a virtual

machine in ec2 and you can create them yourself you can buy them or you can retrieve them from the community marketplace and so this was one that was publicly available on aws in the community marketplace if you searched windows 2008 at the time and once you started using it it would actually uh you know be mining monero in your account finally the mandiant talk i referenced earlier gave a really interesting insider threat scenario again go to that source if you want to see the full details in this case a fired employee shortly after their final day used their credentials to reach a ci cd server created new users stole additional credentials and eventually deleted production databases

obviously the recovery effort here is going to be huge so that's you know a risk to be really concerned about having surveyed that wide range of cases we're going to kind of blow through some trends this is not scientific this is qualitative more than quantitative these numbers are not representative just a bunch of caveats there but quickly looking at it before looking at our cases i wanted to point to mitre attack aws matrix miters and knowledge base of tactics techniques and procedures they released metrics for cloud providers fairly recently but they're planning to merge these into a single matrix soon the most important category for us is initial access where you can see exploring public-facing applications

trusted relationships and valid accounts are the three main techniques and now looking at our data we saw single instances of an unknown initial access vector as well as malicious ami we saw a couple instances likely tied to metadata access three each of abuse of valid credentials vulnerabilities in apps and s3 global right and valid credential theft and again i just want to highlight there are far more instances due to s3 global right than our cases indicated and that could be true of other classes of vulnerability so in the last few minutes here i'm going to do a speed run through the most the best controls the recommended controls for the common vectors we've talked about

here so first up metadata access uh metadata access generally requires some sort of vulnerability to be exploited first so if you have a secure software development life cycle if you're doing pen testing uh stack analysis etc a general code hygiene will prevent those initial vulnerabilities from being exploitable and the metadata service imds is the med data service has a second version that uses a put request to create a session token that you then need to use that token to get the metadata and this will defend against ssrf getting you directly metadata access s3 global write has been a big problem and so aws released access analyzer for s3 that will directly tell you whether any s3 buckets are configured to

allow internet access or access to accounts outside your organization so that's a huge win there and malicious ami is similarly straightforward make sure you're using trusted sources for your machine images the convenience and cost of the community marketplace is appealing um but you know as that case showed uh and especially the fact that the malicious image lasted in the community marketplace for five years you just shouldn't be using the community marketplace for amis for production application vulnerabilities aren't super cloud specific generally um you know asset inventory and patch management you need to know what apps you have and you need to make sure that you're at least defending against known vulnerabilities in those applications through patch

management cloud specific is external exposure so most services and resources in aws and in all cloud environments can be publicly exposed but shouldn't be us keep that in mind valid credential abuse is a big one this is where former employees or maybe third parties are abusing their access so step one off-boarding make sure you have an off-boarding plan in place and you activate it as soon as someone leaves your organizations there's no access lingering there watch out for third party risk follow the principle of least privilege and minimize access in your environment in the first place and then log in monitoring is important and these are some of the heuristics for login activity that are common to use

you know check what times a day location activity and look for deviations and then there is valid credential theft this has been maybe the most common reason is someone somehow gains access to valid credentials and uses them in your environment follow i am best practices with mfa key rotation and avoiding static credentials where possible and again this is a huge topic i'm just glancing over you know the principle of least privilege helps here there's a really good tool that was released this year called cloudsplaining cloudsplain will actually break down across your environment uh in iam whether there's privilege escalation or maybe over privileging and so you can use this open source tool to audit im in your account

and again these slides will be available so you can kind of go back and grab these links if you want to know more awesome so that was a bit of a speed run thank you all for uh seeing through with me um i'm paying the puppy tax with my eight week old puppy so uh there's that feel free to track me down in discord or on twitter if you have any questions um but otherwise these slides will be up on speaker deck and i'll drop a link in the discord channel for this talk where you have access to those so thanks

everyone you