Hey, You, Get Out of My Cloud – Real-World Cloud Security

All right, we have a couple minutes here. So, try to get rid of the nervous energy. I do not speak for a for a living. So, this is I I do this for the challenge and for the uh the opportunity to to kind of expose myself to a little bit more of this this type of thing. Um let's see. I think we can go ahead and get started. So, um just a couple things, just a little bit about me. My name is Scott Arvisth. I put a QR code up here. It's not to hack your phones, but it's uh it's there to provide my LinkedIn profile if you want to know more about me. I'm not going to go into

it here, but that just gives you a little bit about my, uh, career experience. So, um, a couple of things is when I was putting this together, I thought, "Oh, yeah, 25 minutes. I can cover the cloud really well in 25 minutes." And as I started to get into it, I'm like, I need to add this. I need to add this. I need to add this. And, uh, it be the cloud is so incredibly nuanced. And so, uh, there's so many different ways to do things that I ended up just taking a whole bunch of slides out. I'm still taking slides out this morning because I'm worrying about getting it done in the 25 minutes. So, I

took the the title, the inspiration for the title uh is based on a Rolling Stones song. Um, one thing about the cloud, uh, when you're moving into the cloud, I guess, or any environment, right, you get you get a little bit tired maybe at times and maybe you kind of take back and you you relax and then like the like the lyric says, you wake up in the morning and you got parking tickets parking tickets all over your cloud or or issues you got to deal with. And I think that's definitely this the same as in the cloud as anywhere else. So, um, at the outset, let me just say too that I'm an AWS fan. I've been in I've

been inside of AWS for probably close to a decade now. Um, and if I had to choose between, uh, being in the cloud or a data center, I would I think I would definitely choose the cloud because the cloud gives you a lot of things like resiliency, a lot of tracking like actions that occur in your environment. Um, it gives you awesome tooling as well out of the box. So, uh, I'm not trying to do a dig. I'm not I'm intentionally not trying to dig on any cloud providers or anything like that. So, I just need to say that out at the outset. Um, you know, kind of going back to my early days, you know, my early days, I worked

in a at a a company that did bulletin board software. For those of you know, that was phones and dialup and modems and all that sort of stuff. And you used to have this very simple perimeter, right? You had the network perimeter and you had firewalls around that and of course you had behind that you had VLANs and everything else like that. It was a nice simple perimeter though I like to say and then along came cloud service providers and let's just say the perimeters changed. Um like I said the cloud cloud environments are full of nuances. They're full of different ways you can do things and so um I'm not going to give you the best advice here

but I'll give you some advice from my experiences. So, um, going beyond from the simple network perimeter, I like to kind of think about different cloud parameters. And I've kind of got four main ones. And I I guess you could say these applied previously, but I think they they apply more even now. Um, you've got your typical network perimeter, and that's basically the IP addresses that can reach your cloud services or your cloud accounts and all those sorts of things. um they have VPCs, they have services, they have resource policies, all those types of things that that go to that network perimeter. The other important probably the most important perimeter or one of the most important parameters is

the permissions parameters. That's the the ability for people to act or have permissions to act within your cloud. And we're talking about things like um people, organizations, the services that run in your account, um the deployment pipelines that run in your p in your account. Um, basically we want to allow those we want to allow those trusted entities and we want to block untrusted entities. And I think the other thing that uh the other perimeter we have in the cloud is the resources perimeters. And these are the resources that can exist in your cloud. In AWS, you can have things that have their own resource policies applied that are totally independent of your AM permissions. And

so you kind of have to look at your resource perimeter there. So you could have resource policies that grant public access or grant access to untrusted entities and those types of things. And so that's definitely a perimeter. And then lastly, I kind of like to think of the data perimeter. And this one kind of envelops all of the other three, but this one is a focus on of what's your critical data and what do you have to protect your critical data. Combining all of those things from your network, your permissions, and your resource resource perimeters. And they kind of all fit into that. So network perimeter, nothing new here. Um we have VPCs, we have services. Um

it's a little bit more diverse than you have in your environment. Um with AWS services, we talked about resource policies can grant access to your cloud environment. Um outside of your network policies, you have certain resources that are not attached to your VPC. We'll talk about a few of those. Um and all of these things are part of your network perimeter. So again, I I the main point is the network perimeter is a little bit more diverse than it was previously. So I think everybody's familiar with uh your VPC perimeters um when we're talking about AWS. Um you have internetf facing um resources. Those are usually put into public subnets and then you have things that are put into private

subnets that um are not necessarily internet exposed. And then we have all sorts of other perimeters that we have internally. I don't even have time to get into it, but we can there are certain things that we can do to um separate the internal parameters as well from a VPC standpoint with VPC pairing, transit gateway, security groups, knackles, etc. and so forth. Um just a couple things. I I work for ancestry.com and so uh a lot of my experience, a lot of my my presentation is kind of be going to be focused from a web application perspective. And one of the things that you can do whenever you set up uh some sort of web application

website is you can you can block your ingress into your account. You really like we said we only want trusted information into our account. One of the things that you can do is you can apply knackles at at on your public subnets that say hey I'm only going to talk to my web application firewall. And then you're blocking at a lower level. you're not blocking the security group which is higher up on the on the stack and you can basically say hey I'm only going to allow things that things that have been scrubbed through my web application firewall or whatever else you may have out there to protect your your site and you can block uh prevent things that way

at the knackal level you're a little bit more protected from a DDoS type of attack as well um because like you are lower down in the AWS hardware just a little bit of information there so we have the inbound taken care of let's say for example but what about the outbound right often when you look when you apply an AWS security group you look at the outbound rules and they are always the default is outbound to everywhere right and so you could have instances in your account that may initiate outbound calls to the internet you know how do you have how do you protect that perimeter so one of the ways you can do and this is just one thing you can do is

you can set up your routes you can set up transit gateways I'm we're not going to go into that but basically I can set it up that all of the traffic that's originating from my account going outbound to the internet ends up at a network firewall somewhere. And then that network firewall I can I can set up sort of cotta rules or whatever else you have that basically can say I'm going to block connections to untrusted networks. I can block a TLS. I can block SSH. I can block SFTP. I can block a lot of those things using a network firewall. Right? So, I've got my ingress and my egress completely covered, right? Well, not quite so fast, right? We control all

the outbound traffic, right? Not quite so fast. What if something in my account goes, "Hey, I'm going to take I'm going to I'm going to make a call out to the internet. Let's There's these things. I'm getting a little bit ahead of myself. Apologize. There's these things called VPC endpoints, right? VPC endpoints. If I put them in my account, they they basically create an AWS service to be a local call. So I can make So if I'm calling the S3 service, I'm calling it as a local connection. It's not going out through the internet. So you have a little bit of a hole here, right? So u this is this traffic since it's going through a VPC endpoint, it's

not going to hit my network firewall. It's not going to be blocked by my security group. So what can I do? And that one of the things you can do is you can put a VPC endpoint policy on your networks that basically say, "Hey, I'm going to only allow S3 traffic out to trusted locations." And this just gives one very simple example. I have this uh resource account that says, "Hey, I'm only going to allow me to talk to buckets within my account, but I could also set that to be my organization, or I could also set up several other things, but this is just an example here." So, we've covered all of that, right? So, so now we're covered, right?

Well, not quite because like we said, a lot of different AWS um services have resource policies that are completely unattached to your VPC. Um just to get an idea of what all of these are, you can look it up on AWS. Um here's a link, you know, but basically you can see resource-based policies and if there's a yes there, this is something additional you have to work worry about. So, one example that I'll just, like I said, there's lots of there, but one example that I'll talk about is just API gateways. API gateways are basically an endpoint that can be publicly exposed. It's not attached to my VPC. I don't have security groups on it. How do I control that? Additionally,

that that API gateway may have a lambda attached to it. That lambda may have IM permissions into my account. That's also going to be an issue. Is there authentication? Is there monitoring? Do I do I don't have any monitoring there unless I've set it up on my VPC endpoint. Also that that Lambda itself doesn't have to necessarily be ta attached to my VPC endpoint or to my VPC. It can I can configure it that way but it could also make outbound connection. So I said the devil's in the details. There's lots of nuances in when you talk to the cloud. If you look at the API gateway um policy itself, what's the default behavior? If you look at the

decision tree it comes in is there resource policy no you would think the default would be to block but in the case with API gateway is it's allow it's allow everything right and so um there's definitely another network boundary into your account that you have to worry about we can fix that really easily be by putting a a policy on my API gateway this one's a REST API gateway that basically says hey I'm going to only allow traffic from my networks this is in a security group but kind of gets into that resource policy space. V2 API gateways are a little bit different as AWS is. Nothing is consistent in AWS or in the cloud environments. And so B v2

you may have to do other things like disable the VP the API endpoint and other things. I said there's lots of nuances. Um I can't go into other things. A couple other things that don't you don't really come to mind when you're thinking about IPbased policies into your environment. Uh if you read if you've read any of the news for any sort of cloud environment, there is these things called API keys that basically give you the ability to do anything in the account from anywhere on the internet. And if you look at the default sitting on those API keys, of course, they're allowed from anywhere from by anyone, right? Anybody who has them. And there's also been plenty of stories um

like other AWS services like S3 buckets that have been publicly exposed. And again, that public exposure has led to issues or embarrassment for those organizations. So again, whenever we're thinking about resource-based policies, here are two really important ones that I'm calling out besides API gateway that, you know, I can put on all of the keys that I I grant access that I can make sure that those are only allowed from trusted locations. Since we're talking about networks, I've given an example of IP addresses, but there's other things that you can do like trusted accounts, trusted other things. But I can put a resource policy on my I API keys. Same thing with my S3 buckets.

Um, I can put a policy if it needs to be exposed. I I can put a policy that only allows it from specific IP addresses or in the case of a VPC that it's only allowed from a VPC. Uh, just two example there. Another thing is there's some unicorn services in AWS that don't have resource policies that don't have security groups associated with them and these are yet another thing that you kind of have to worry about from a network perspect perspective. AWS has a service called transfer service which is SFTP. Um you may or you may not want to um block who can who can reach that service. Um, there's also the Cognto service which is AWS's own identity

provider. There's AppSync, there's CloudFront, there's other things here as well. A lot of these things you don't have either one of those options. Some of these will work with the AWS WF and so you can put some network parameter network perimeter around those those services as well. But the point is there's lots of different ways of um network exposure, public network exposure from an AWS account and you have to take all of those into consideration. So next thing is is per permissions perimeter. So we're going to leave network alone. We're move on to permissions. Right? In cloud accounts, everybody or is going to need access some level of access into your environment. That's just the way it

works. Back in the old days very few people got into your data center. The old data centers you had very specific people. They there was all sorts of control around that. We moved to the cloud and we basically grant everybody access more or less or may need to access into our virtual data center. And so this is probably one of the more important perimeters that we need to think about in the cloud. Um there's many stories that have talked about people checking in um access keys into GitHub or or um trust policies being weak or other things like that that causes all sorts of issues. The famous story that I always bring up is cloud spaces who had uh access into

their cloud account get leaked. Uh hackers got hold of it. Uh they said give us a bunch of money or we're going to shut you down. They said we're not giving you any money. and they hit their they hit their automation script and it just started deleting stuff and by the time they got it stopped they were out of business. And so the permission permissions perimeter is a huge huge important thing to worry about. But inside this I want to break it down that there's several other permission perimeters that we've got to think about. It's not just access into our cloud, but we need to think about how we're going to allow humans into our

our environment, how we're going to do our deployment perimeter, how we're going to do deployment, and how we're going to separate that from the human, right? And then also the service roles, which is like once we're up and running, those service roles, how are those those perimeters need to be managed well inside of our cloud account? So, first of all, human access. So, a couple of ideas when it comes to human access into your cloud accounts, right? My my my take on this is anybody that's going to have access into a stage or production environment should be readon at most. And may I may even want to control what readon access they have, right? The other thing I want to make

sure is that they are um they are limited time specific, right? They're not permanently granted access. And so I can use like like the STS service to to use only temporary tokens so that I don't have to worry about access keys getting linked out leaked out. The other thing that I think is really important is role chaining. We'll talk a lot about this in a little bit but role chaining is a big risk within um the permissions perimeter um that we've got to worry about. Another I think another good practice is when you're talking about creating roles or or users or other things that you name them with a specific startoff string like a user dash or something else like

that. So when you're talk when you get into some of the more advanced topic like service control policies or other or IM based policies you can use that to determine between if it's a a user a human user or something else. Um, other important human parameters here at this level, I I won't talk about it that much, is administration roles. Really, assuming if everything went bad and if you had one of your administrators get had their uh their laptop get compromised or their permissions get compromised, right, a good thing to think about is is can we can is our cloud environment still safe? And so I like the idea of taking a ring zero approach

that says, hey, if we had some sort of compromise that those AD credentials aren't going to going to be a weak weak point in our environment. So we'll leave humans alone and we'll talk about IA deployments. I think a critical thing is the goal is I don't want humans to be able to do anything in my production environment besides see things, right? And a big crucial point of that is I have to have a really strong IA deployment permissions perimeter, right? So this is going to be what's going to be allowed to deploy things into my account. Um I think these are this is probably one of the the crown jewels, if you will, of your environment, right? It's

really critical to protect um your your deployment pipelines into your accounts because if those get compromised, those have those have all the permissions to create things, delete things, do things, all that sort of thing. You may want to separate your uh deployment infrastructure u between different types of trust in your accounts like a high trust like a PCI account versus a corporate account versus something else. So these are all perimeter are all things to think about in your IC deployment perimeter. There's lots of things here that can go on. The other thing that I think what we're going to talk a lot about here is uh the deployment pipeline is one of the places where we can put something what I I like

to say is security gates. Things that I can do in this per permissions perimeter that can block risky things from being introduced into my accounts. So things like create blocking public exposure. We talked about that all the different ways that that can occur just previously. Any sort of escalation of privileges, that's a big risk. um unauthorized services or regions um uh overprovision resources that have too many permissions or they're granted resource star. Um and then the last permissions perimeter. So we'll leave that one behind for a bit. The last one is the services access. So this is your steady state what's running right and um it's really important as we talk about our permissions perimeter.

These are a lot of times you may have a dev environment, a stage environment, a production environment and those permission perimeters should be separated from each one of those um I guess you could say um stages or um environments. Um and another thing you want to do is make sure that those permissions what are running steady state that they don't have they're not provisioned to have access to all resources. Um the other thing is we're not going to talk about a lot. I'll just mention it here is IMDSV2. All of those all of those services are going to run with permissions. Those permissions uh let's say if I have a website my website has a

weakness vulnerability. Someone finds a ways to go in and dump the temporary credentials from that service and be able to access them and run those. Right? If they pull those out and they try to run them and they try to test whatever they they can with those permissions, I can't they can't do that. If I've set up IMDS and I've set like the hop count, that's a specific thing to be one or two. It says basically AWS says says I'm sorry that's not being called from the account, it's not going to be allowed. So there's lots of things to think about here and there's more that I could add those, but these are maybe the top three. So a couple of

things about breaking the permission perimeter. So uh and we'll talk about how we go about protecting that. So IM trust policies here's one thing right if you go look at your AWS accounts um every every role has a trust policy associated with it. Um in this case I have a trust policy and I have an account 333 444 and uh colon root right and what that essentially says is any role in this account can use the the permissions associated with this policy. Well, do we know what that account is? Do we know what the the role can do? Um, this is something very very important to watch, right? If this if something were like this were to show up in your uh trust

policy. Also, another thing, you know, developers are very crafty people. I would say um they may do things that say, hey, that I may get into a situation where I may need to jump in and be able to to debug my code. I'm in a tight situation. I may need to debug my code. And so you may see things like chain roles showing up in your trust policies that basically say I'm only going to use this for break glass, but I'm going to jump between different different roles until I finally get into the role that I want and then I now have permissions to do that. So here's a per there's two good examples of how this

permission perimeter can be one of those perimeters can be broken. Here's an example of a ch that what I'm talking about roll chaining is I start out in a boundary account. I go to my development account and then I switch over and I assume roll into my production account. So that's the type of thing I'm talking about. Other things that can show up in in policies that break these these perimeters is I there are several different ways you can escalate privileges within AWS. One of the common ways is IM pass with resource star. What that basically says is, hey, I trust this role to start up another service like a lambda or an EC2 and then pass it

any role that is available within that account. And that's that's another thing that breaks that permission perimeter boundary. There's lots of other things like attach a role policy, create user, create role. There's a whole bunch of things. I can't list them all here in the time given, but I've got some links to those as well. Um, and then another one that I'm not going to go into a whole lot, but confused deputy. This has been another problem where essentially I have a role that role trusts an account that that that account also has other accounts attached to it that are not trusted by me. But there's a way using the confu confused deputy attack where a role in a

completely untrusted account can basically assume a role in my account using the the confused deputy attack. I don't have time to go into it here, but I've got a link to that as well. Again, another way that our permission perimeter can be broken. And then we talked about unmanaged services, right? Um managing your identity providers as well. AWS Cognto can do what is this this is my my slide here. Let's say AWS Cognto is again it's AWS's identity provider, right? And there are certain ways that you that you can well you can set it up to have its own identities associated with it and then you can look and you can set up the trust policies

within your account to trust that co cognto identity provider. So there's another way that uh those this permission boundary can be broken is if I'm if I have unmanaged services like cognto that are allowing people to authenticate to my account and those authentications can then assume roles into my account as well. So fixing the permissions perimeter um making sure that identity provider that we use to get into our cloud is not public. I think all these things are no-brainers, right? it that that you can only access our our API the APIs in our cloud account or the console in our cloud account from a trusted network that it requires multi au multiffactor authentication excuse me to get to that

point and then that any sort of access there is is time bound uh API access keys I think they're the devil they should be avoided um unfortunately some things sometimes those need to exist and then we talked about trust policies we talked about assumeal trust policies, but you can also put other identity providers into those trust policies. And so that's something to monitor, looking for cognto or other types of things showing up in those trust policies. All right, so um let's talk about some ways that we can control some of these permission parameters. Don't have a lot of time to go into them, but there's things called service control policies in AWS. These are things that I can pro I can set up

at my organizational level and I can block certain high-risk things from being introduced into my environment to help protect those those perimeters there. Uh recently AWS announced resource control policies which are more tied to the resource. Um but those can put put boundaries in place as well. And then the other thing that we're we're going to talk about is those like I I talked about the deployment pipelines. It's very important to have deployment controls in there that can block some of the the um high-risk issues. Um some example of surface control policies that I can do like I said I don't go into the details but I can block unapproved services unapproved regions. I can block

DNS changes that may be a real big issues that I need to control. I can block the creation of resources in public networks. I can block some escalation of privilege actions. I can block creation of access keys etc and so forth. Um I can block changes to any of the network infrastructure we talked previously. Uh security gate rules in our deployment deployment control. Uh just examp you know a lot of times we use cloudformation template or we use terraform to deploy those in our deployment pipelines. We have quality gates we should be building in security gates as well and some simple security gates that I can put into place. Here are just some some highlevel ones that

we can put into place to block some of these uh to to prevent some of these um uh boundaries from being broken. I don't I'm not going to go through all of those, but those are an example. All right, let's do a time check. Okay, I'm running out of time here quickly. So, um resource perimeter. Um we've talked a little bit about this. A poorly implemented resource perimeter could create a public exposure. We've talked about that. uh role should not be able to create uh external resources. Uh there's two things to worry about here. Some resources are protected by a resource policy, but as you can see, there's a whole lot of nos selected in that

column. And so resource level permissions are done in IM policies. Those we may we may have to do put protections in in IM policies. The problem with those IM policies or RO policies is you can do a really good job with IM roles and then one RO and everything's least privilege and then one RO gets in there and it's full privilege and it has access to the resources. So that's a little bit difficult. So breaking the resource perimeter, the resource policy put on in this case a Q pol SQS a Q res a Q service. Um that basically is granting anybody authenticated to AWS to fully manage my queue. Uh I call I call this a

resource perimeter violation. Um I can put I can create a role that has a permission to do a dangerous action like terminate instances with a resource star. All of a sudden, if this role has an issue or gets compromised, I lose all of the compute within my account. Resource policies, we talked a lot about these, right? One of the things that you should do is turn on uh public access blocking at the organizational level. That will create all your that will make sure that all your S3 buckets are not publicly exposed. That should cover you, right? Well, not quite. If somebody creates a resource policy that trusts a specific account um that maybe you don't know

what it is and maybe it's doing get object uh here's a way that the res resource perimeter is is broken. Right? I'm now allowing a an untrusted account to get objects from from this particular S3 bucket. This is a policy put on an S3 bucket and the block public access didn't necessarily prevent this from happening. fixing the resource perimeter. We talked a about this, right? Turning on public access blocking, resource control policies. We talked a a little bit about that. And we talked about the security gate that can put standards on what how resources can be configured. Um what sort of what things are allowed to be uh what things are trusted to to call the service and

what actions can be made as well. Um, I mentioned this is new. It resource control policies do not cover everything. There's just five services. S3 STS KMS SQS secrets manager. Um, and again, here's some examples for some security grade rules to block that. All right, I am really short on time. Uh, so data perimeter is the most important perimeter. Essentially, this is looking at where is the critical data to your your business and that you need to know what to run. And we're going to go through all of those things that we just talked about, the network perimeter, the resource the the permissions perimeter, the resource perimeter, and make sure that all of those are in place and they're strong

and that we don't have any weaknesses. They're in place. And then one last thought here on the data perimeter, right, is assuming we had a complete compromise. um our data backups. All of our data, our critical data is protective, right? You know, answer that. AWS has a service called S3 object lock that I would recommend you looking into that can do those types of things, assuming if you do backups and you put them into S3, you can put an object lock on it that nobody can delete that, not even the root account can delete the resources. Knowing that where your data is is half the battle. Um so in sort of closing right um there's a

lot of different nuances in the cloud. There's lots of ways things can go bad. They can be broken. There's lots of different perimeters to think about and to to quote Tolken here, right? You have to assume that there's a dragon out there somewhere that's going to try to break into your account and do bad things to you, your organization, your co-workers, your teammates. So don't leave that out of your calculations. So thank you. Thanks for coming and um hope you had a great conference.

Thank you. Thanks.