The Dev Ops Crusades

Dave's vault welcome to the DevOps crusade the causes and casualties of AWS misconfigurations here's the ground that we're gonna cover since this is a security talk I'm going to start off with some victim shaming and then after that I will go into a two-ish minute rant on why I blame DevOps for security issues and that is followed by the technical side of the talk which nobody remembers where I explained the complexities of permissions in AWS and I will also go over some tools and tips on how to assess your your account your account security in AWS for free and then we'll conclude with some steps on how you can lock down your AWS account oh and if you

happen to use a different cloud provider please stay tuned for the end because the tips I give will be ubiquitous no matter who you choose tell me if you've heard this story before a minor mistake led to a relatively massive leak it's not the first time we've been here has it we've had s3 data leaks affecting millions of users every 6 to 12 months for the past three years this is ridiculous especially since company a's can save billions and the millions of us who use these services and products can save lots of headache and misery with 17 lines of json I'm talking about curly brackets key value player and key value pairs is here but I want to go over the

2019 Capital One breach I want to emphasize that it is different and the reason why I want to go over this one with you is because it's not the usual open s3 bucket out on the Internet the folks at Capital One weren't stupid they they had it LA they had what they thought was locked down behind a web application firewall that was all that also just happened to be misconfigured the attacker was able to get through the firewall onto an ec2 instance and then connect to what I think multiple s3 buckets containing millions in customer data my thought is why that particular ec2 instance why did it have that much access to s3 to download that much

content that's the question I that's been buzzing around in my mind and the other question as follows what can I do to prevent this from going on in my own environment but first I want to blame my problems on DevOps I blame DevOps I blame country company cultures that prioritize pushing features at breakneck crackhead speeds I am we're building things fast and they're breaking down even faster but here's the thing the devops crusade it's over day one we are all we're all taking the DevOps kool-aid by now all of our we're moving our infrastructure to the cloud we are just screaming Automation at the top of our lungs we drank the DevOps kool-aid and we're paying for it of

course there's another buzzword solution to buzzword problems I hear this bounce this term bounced around a lot dev SEC ops I don't like it it's bulky it's sell elegant and it's just throwing security in the middle of it to try to get curity people on board with this but part of me thinks to myself you know why stop there I mean we're all tea on the same team right I mean come on SEC security isn't something that you throw into the middle of things it's something that needs to be ingrained into everyday business so the DevOps I'm arguing for values genuine dialogue and communication between development security and operations we're not talking about just throwing around code faster and

automated fashion between the teams and between the silos what I'm arguing for is that every developer be trained to understand the security and operations requirements so that they can write secure code that works that's the that's the vision I want to sow to and part of that is enabling the developer and neighboring developers to write that code and in order to do that they have to understand the complexities of s3 so what's so complicated about for access management and s3 well here's the details you've got two options and I say options in the same way that seatbelts and airbags are options theoretically you can use one or the other and feel relatively safe but let's

use both the first system is I am roles and policies I am has a lot to do with users and the users and entities acting on those users behalf the second the second system our bucket policies and the bucket itself determines who it will open itself to for its resources now I am roles here's an example for I am roles as a developer I have access to source code for our ongoing projects for secure ideas as a consultant with secure ideas I also have the I also have the ability to just spin up a cracking server password cracking server when I'm on an engagement so I am roles are specifically associated with users or entities acting on users behalf and

there have written statements attached to them that determine what they can do and what they can access that's what that's I am its users its user or service centered and I say service centered because a dus pipeline has a role within itself it's not necessarily human you wouldn't call it a user but AWS code pipeline takes code from any source that you give it whether it be s3 or github what builds the code and then to pull and then deploys it to whatever wherever you want to do values it to wherever you want to go so that code pipeline has to have a role that allows it to get to grab it from the source and

deploy the code it those policies have to be written into the code and we refer it to do this job and to do its function now for s3 bucket policies there's a lot that we can go into but I have to I have to cut myself off because we only have 20 minutes together and so one things some things I will mention you can lock down with dick bucket you can lock it down with multi-factor authentication you can restrict the level of IP addresses that can access files on this s3 bucket there's so much you can do but if you understand and know I am roles those same principles apply to bucket policies so mastering so mastering permission

structure and s3 involves mastering I I am roles and mastering s3 bucket policies but insecure use is three easy clicks away now this is beautiful its elegant it's what we saw it's probably what the CEO was sold on when you when it was brought up to move your infrastructure to AWS it's these types of things step one and this is a step by step process on how do I connect how do I access s3 through an ec2 instance so step one you create the I am role step two you attach the s3 full access policy this is built in this policy use wide open access to s3 is built into every single AWS account and takes or clicks

to use then you just set up and launch an ec2 instance just spin it up and just click and through and follow the on line wizard now we go into how the secure use the right way to the right way to access s3 from the need situ instance and I'm not going to lie to you this is complicated but follow along with me if you will first you create an I am role then you create and edit as custom s3 policy which involves you learning some JSON memorizing some of those you value pairs some of the actions that you're allowed to take on Amazon s3 we attach the custom we attach that custom policy then we set up and launch the ec2

instance spin it up and you SSH into it you're in the Box you use the built in AWS CLI and access denied it doesn't work you can't chain you can't change the rules of ec2 instance while it's running so you have to tear down that ec2 instance go back to step two and continue this process over and over again this secure use of the platform is difficult and tedious and requires a lot of trial and error this is not something you just hate you just throw at the security team well it's three months in production and say secure this for massive web applications that are cloud native it could be this is the gist in

the interaction between two services regular full-blown cloud apps could have dozens of services that interact in different ways and so this process gets exponentially more difficult the larger the app is and the more that you leverage Amazon Web Services so with that said how can we assess security on AWS resources well the one tool I'm going to show and present to you today it's freely available all for most people working on AWS is a tool that will cost you nothing no matter how much you use it it's not a free tool and then you get a bill for $400 next month and you go screaming at your devops guys like what are you doing it's absolutely

free and it's accessible to anyone who uses it and with that we switch over we switch over to the live demo portion so here's an example here's an example of a bad as bad ec2 server you could see two servers roll oh one second yep so this is an example of a bad that's three yesterday demo roll it only has one policy in it and you note the use of wildcards it's just four lines simple elegant beautiful right but if we go into but if we go into s3 and we actually look at all of these select all of the app options that it gives us everything is allowed green means go for everyone including you the legitimate

user and an attacker who just happens to acquire the role so some of the so some of the permission structure or that you can get with full s3 access is ridiculous you can list all the you can do things like list all the buckets download you can delete entire buckets if you store logs in s3 or attacks or malicious activity someone with full s3 access can go into the logs and race their traces there are multi room any things your way to many actions here that could be abused so let's tone it down to something more sane so here's an example of a more secure s3 policy and now this is something now this is the 17 lines of

JSON that I'm talking about you're keeping it saying you're designating you're designating one bucket that happens to be some bucket and the objects within that one bucket you're allowed to access and the actions that you want is the simple use case of s3 when you think of s3 you think I want to download files I want to upload files to some server that I don't want to manage and I want but I would like to refer to the server by name so that is yet so that's get object code object and the list bucket operation now when we run them now when we apply this and run the simulation again access denied and this is a good thing

that means someone that means someone who has attached and has access to this role can't really you can't really use this particular role to abuse use it on multiple buckets on orbit and arbitrary bucket what they can do and we can and we can go into death I'm going to specify the object by using the AR n by just copying the air and

ultimate in hand-eye coordination and minimal and I'm also going to specify the bucket by cop by copying the air in as well and all I'm doing is hey let's rerun the simulation rather than on star wildcard any resource here let's specifically designate one particular bucket or one set of objects here we're using wildcard thoughtfully we're using it specifically to refer to every object or every file within that bucket rather than all buckets are all resources throughout your AWS account now you run the simulation and it's the same thing right well when we go into lit and when we go to these specific permissions that we're going to we see that these specific permissions are allowed so you

can list bucket meaning you can refer to this bucket by name which is kind of needed if you want to access the files got in at least know the whole folder name right get object with get object which get object which means you can download files and put objects which means you can create and upload files to s3 and that's the demo portion so with all that in mind and all that said whew wrong screen wrong screen there we go

with that said how do we lock down our agent how do we lock down our AWS accounts first we need to understand the secure DevOps requires review and allows us of the code and the cloud what that means is is if your developers or rut or DevOps people or whomever is writing is writing and making this weird black magic make sure that you review what they're doing not just code wise but also in AWS have those changes mentioned in the pull request and then bake this into the process of development you can also use cloud formation templates and to have the cloud infrastructure embedded into the codebase term that gets thrown about is infrastructure as

code it's really effective to help get eyes on on the changes that are being made and then make sure that you actually have the reviewer check for security issues at every pull request number two resources need to be isolated and given the least possible privileges so one easy way to do that it's just to have two separate AWS accounts one for production that's cut that's nice beautiful shiny product that's available for the customers and then the other for the half-baked works in progress so the devs mess up here they're not going to expose confidential data we understand that isolation is needed for code it needs to also be isolated for service infrastructure that has a price tag

attached also make sure you that you have both that you be suspicious and weary of wild cards and not use full access permissions do not allow full access permissions number three understand that secured you a patches are bug fixes security improve our features and they need to be treated as such have the devs use tools like the AWS policy simulator to debug permissions and write code securely the first time around and please please for all that is sacred do not throw your massive cloud native application to the set team and tell them to lock it down after it's in production for three months while the fires are that we there are enough fires to put out this this dumpster fire can

be prevented now using all the techniques that we've discussed we could prevent these headlines if the attacker got access to the ec2 instance and it only served a Maya role and only had access to one bucket we wouldn't have this massive leak affecting millions of people secure dev can create specific policies that save billions of dollars and save a lot of us a lot of misery security is a never-ending campaign and this sort of IT technology journey we're all trying to build and create beautiful things and change the world I believe we're all on the same side and we shouldn't be fighting against each other security and the concerns with that is just a never-ending campaign within that

journey so my invitation to you is let the devs join in on the action they are eager to help you trust me you should see it when you should see it when we actually teach them that they can hack their own apps their eyes just light up and it's just like

that's my invitation that's simply my invitation to you so invite the devs on the journey let's the devops crusade continue let's retake the cloud Davis vault [Applause]