How to take over a production system in the cloud - Paul Schwarzenberger

hi good afternoon as it is now afternoon and I'm going to aim to finish in time for lunch so I'm Paul Schwarzenberg a cloud security architects and SEC ops so I'm going to talk about how to take over a production system in the cloud so this is based on some of the work that I've done with different customers over the last four to five years so before I start how many of you are working on cloud systems in some way or another this is probably at LeapPad at half but probably about half of the audience and then of course there's some who are but didn't want to put their hands up so that's fine so what I'm

going to do in this talk is I'm going to give three examples of cloud security issues and these are all issues which I've encountered as I say working with different organizations and I'm going to aim to do some well am I am going to do some demos and so we'll see live demos we'll see how they go and then what I'll then discuss is you know why those issues happened and then talk about how we could prevent them and on the prevention we'll look at not just sort of technical ways of preventing these types of issues but also sort of organizational cultural skills and training needs so we look at that as well as the technical aspects so the

first issue that I'm going to talk about is relates to identity and access management roles so so this particular issue a demonstration is is with Amazon Web Services AWS the that's not to say that AWS is any less secure than other cloud service providers not at all but I chose AWS because they are the market leader and in fact there was a gardener Lake Cisco of a Magic Quadrant came out yes today or the day before with AWS at something like 20 billion revenue billion dollar revenue per year compared to the next the next one which is the zero at four billion per year so they are by far the way the market leaders are guessing or

the people who put their hands up probably a little quite probably the majority I'm guessing it probably AWS so so what I'm going to do is to because I'm sure some of you are very familiar with AWS some of you are probably not at all so I'm going to give a little bit of an overview into the context before then showing how how to actually take over a production system so what I've done here - so I hate to disappoint anybody I'm not really going to actually take over a production system like what I am going to do is to simulate what actually happened so okay so first of all let me just give a bit of a bit of a sort of

introduction just a little bit bigger just so so identity and access management is AWS is as the name implies it's their service for controlling access and authentication and granting rights both to individual people and also to any other services computers or whatever that needs to do certain things within an account so this is just an example so this is my account and or one of my accounts and you can see here that within Identity and Access Management you can firstly define users so you can see here I've defined a whole load of users you can define groups so I've got various groups associated with groups and then there's also the concept of policies and as you might expect

policies give certain rights to do certain things within AWS so that might be permissions to spin up a virtual machine or it might be permissions to create a new user or permissions for you know elects for business device set up so they can be very granular or you know very broad and all of them can be defined either through a GUI or in code through Jason so hello the policies and policies as you might expect can be associated with other groups or users they can also be associated with roles so roles is another concept so roles the idea of a role is it's is it's a way of granting right to a service principle it could be

a user could be a group could be a server could be another AWS service and when a service principle assumes a role then according to what policies you've set certain policies will apply to that role and so if we have a look at a role which I'll just choose fairly randomly okay cloud trail cloud watch log roll so there's certain permissions associated here with policies and the whatever is assuming that roll gets a secure a session token which is valid for a certain period of time in this case an hour so that's sort of how all that work so that side ends in Access Management so that's just sort of give the context so one of the organization's I was

working with I sort of came in as a dev sir corpse cloud security architect and when I came in at the point that I came into the project there was already a production system that had been going for about a year and that production system was used for various web applications and although there was a strong security department within the organization there hadn't been any real sort of detailed security analysis of the code so one of the things that I did I was embedded within the developed team as a security specialist so one of things I thought what I better do is take a look at the code so I looked at the code and one of

the one of the things which what one of the requirements is for some of the servers within Amazon the production servers to be able to do certain things within Amazon like access s3 bucket for example for storage or even spin up other instances and so on so to do that the server needs to assume a role which then has a policy associated to either grant those permissions for certain time typically an hour so I looked at the code right let's look at the code

right

okay so this is a sort of representative represent well this is pretty representative of what I found I just took out a couple of details that might have identified the customer so we the first the first section says what rights they said this is a policy so this policy says at the top it says I've got certain rights to do certain things in this case on s3 so you know you could this actually gives quite a lot of rights because you can list buckets list objects and you can get objects and you can put objects so you could do quite a lot with this three and then you can't do it to all of s3 only certain named

buckets and you know you might think I just did this is a demo and put like bucket - name - secrets but actually that's almost what the actual name of the pocket was it did have - secrets at the end so just to help anybody who was looking for something so yes and what that did is gey give pretty much pretty broad rights to these particular buckets and then as well as s3 it also gives permission for ec2 so ec2 is Elastic Compute cloud so this is Amazon Web Services infrastructure as a service to spin up virtual machines so you can see that actually anything is allowed so that's not good security practice obviously to allow any action with an

ec2 but and also any any resource and then as well as that there was another policy which I haven't bothered copying which allowed full access well not full but act allowed quite a lot of access to route 53 which is Amazon's domain name service certainly enough to create new records and delete records so that was the policy and then bearing in mind that a roll gets to assume a policy so if we have a look at the assumed role policy so you can see here's the assumed role policy that was on most of the production servers so I'm sure we have some AWS people the room has or AWS people have worked with that at least so would anybody like

to sort of have a guess or know what's wrong with this policy or what the implication is of this assumed role jason associated with the policy go ahead pun yeah yes sir the world card yet almost I mean so the principal so the comment was does the wild card the star allow permissions to everything almost I mean the this is this bit is the role what's allowed to assume the role and then the other thing that I just showed you was the policy which is what permissions are so the fact that it says AWS : star actually means that any any other AWS account whatsoever is allowed to assume this role and then gets the policy go ahead well your as

you said a bit it allows anybody to assume this role provided you know the name of the role or you could guess the name of the role it'll become bit clearer when I demonstrate it but yeah so that's the problem so I thought well this is pretty serious because okay the production AWS account is owned by this organization but anybody else in the world could just set up their own AWS account assume this role and then get permissions on this account so thought well this is pretty serious pretty quite a high risk so as you do as the security person flag it is a high risk and then I thought well I better fix it so I

created because I was a better than the team I did the usual process which was to you know clone clone the the repository create a branch on my branch I corrected it and the corrects the correct role is shown here which isn't very different it's only about two lines of code difference it says the principal instead of being a WS : star the principal is service ec2 Amazon AWS comm and the way that a WS interprets these assumed role statements that is not anybody's easy - it's only easy to within this particular account so which was what was the intention because it's meant to be a server in this account assuming that role to do certain things

so that's the correct one so I corrected that and that's one of the easier fixes I've done literally related the fix itself to you know about five minutes and then you know committed that on my branch put it through the CI CD pipeline went through all the automated tests then it got merged into the dev branch and then that was all good so then a couple of weeks later I thought I wonder if that's actually being promoted yet into production so I checked on the production system and it hadn't and so it raised it as an issue and then still nothing happened for quite a while and that was nothing to do with this code it

was just to do with other other factors but it meant that although we would fix it the dead branch had never actually got into production so I thought well this has really gone on long enough need to do something to actually get some attention because even though it's raising things there's high risks and critical risks yeah that was just being unfortunately not treated with the seriousness it deserve so so what I did then was

I would get this oh yeah right right yeah so I did then was I got my own account which I've just shown you so this is my own account and then I'm simulating this sub simulating what I did with AI to simulate what actually happened I've set up my own separate AWS account to simulate the production account and there's no connection between these accounts you know I would happen to both be owned by me but it could be any anybody so anyway so what I did was to I use my own account and then this is what I did so if I just yeah expand that bit again so yeah so so I'm logged into my own account there was

nothing to do with the organization that I was working with so just click on here and then I can do switch role so switch role and then this now gives the switch role option so all I need to do to switch role in and then take advantage of that sort of code is to put in mr. dunn's before the name of the account which is this and then the name of the role which is this and then switch rock ok so now I've got into a completely different account and you might think well that's quite hard to do because you'd have to know the name and the role but on the other hand you guys probably

wouldn't because quite a few you know good good at numerating things and also bearing in mind that what quite a lot of organizations do is to put a lot of their code on to github some of it in public repositories and it wouldn't actually be that hard to find out oh of course the sub had left the organization they you know could have a copy of it the code base anyway so I've gotten to the other account so if I look at so for so yes if I look at users for example so now in the other account which simulates the production account you can see here I don't have permissions well this is what you'd

expect because this is identity and access management and the policies didn't have anything on it identity and access management so now let's look at some other services so if I go into ec2 so in ec2 now and you can see that there's no indications that don't have the require permissions so what I can do here is to go into running instances so there's an instance running so in this case I've just created one instance just as an I give you the idea in the real example there was about 50 production servers so then then I can just highlight this and then I could do actions for example terminate so I could terminate that instance and then if I

wanted to spin up my own instance which might you know under my own control I could do this so I'm just creating a new instance now and do this you know so now I've just created a new instance you know so and then I can view the instances you know so I've just shut down one instance opened created my own new one I did also you know in the real case the there was access policies that allowed route 53 changes as well so you could have pointed dns to my server take a note you know taken customer credentials for example could have mind lots and lots of Bitcoin you know so the possibilities are endless

so what I did in the real case was I didn't actually delete production servers but I did get to that point took a whole load of screenshots put it in the PowerPoint presentation sent it to all the senior managers and then it all got fixed in about three days which was good so that was the first issue so from a technical point of view you know as you saw from a technical point of view all that to get back to this yeah all that was actually wrong was four lines of code so yeah this is just some screenshots of what I've shown you so that's the incorrect one this is the correct one so I'd why does it happen

nobody admitted to doing this they all said it was somebody who'd left it had been like that for a year or so without being noticed fortunately I don't think any attackers noticed as far as I'm aware there's no evidence of that anyway my guess is that the original person the person who originally coded this he you know he wasn't a security person and he probably just googled assume role came up with that code and copied it because that was lower down on the search list I mean that's just a guess but I expect that's what happened you know so we'll talk a bit later about some of the things you can do to prevent people doing that sort

of thing yeah we'll talk about that later so that's the first issue so I'm going to move on now to the second issue so the second issue was the favorite topic of last year amongst clouds and well not cloud just cloud but data lost generally and that's s3 buckets so you know there's been particularly last year and some this year as well there's been loads and loads and loads of losses of data from s3 buckers this particular example is Dow Jones 2.2 million clients lost their information because of an AWS configuration error on s3 so we'll just take a little look at s3 and see how easy it is to miss configure s3 so right

so I've actually done this in my other account yeah we got s3 on my other account right so what you'll see is on this account I've created s3 is simple storage service so this is Amazon's the way that you can store data in s3 in in the cloud with Amazon and again just like the first one this demo happens to be AWS however you know all the same sort of issues are there with Azure storage and you know Google storages it you know anything could be misconfigured so you can see here I've got two buckets I've got two buckets and you can also see that there's this orange sign saying public so which and you can see that

neither of these buckets are marked as public now because of all the data losses last particularly last year Amazon of that they've upped the game a bit well quite a lot actually with s3 so one of the things they've done is they've put this sign that says public and then what what it what it does what the code does is to look at the permissions on the bucket and if it's public then they'd actually be an orange sort of symbol next to the bucket so you can see here that you know and and also it says access not public so they've made it a lot more obvious through the GUI anyway bear in mind that

quite often you know quite often everything should be generally everything is coded anyway so quite often the people are actually writing the code for the don't see any of these warnings because they're just coding you know but anyway you can see here that neither of these are public and then you can see here that if I go into this bucket so the buckets called CPMs 3 and if we look at permissions you can see that the permissions are like this so you can see that ok so my own root account does have permissions which is what you'd expect and then basically nothing else has permissions so the public does not have permissions to list objects to write

objects or read or anything so yes there's no permissions allowed which is why it was not public

so yeah where am i right so we go back to the overview so you can see that I've created a file called password file dot CSV so any of that you working quickly will have already downloaded this farm but maybe not just yet so if I click on this you can see that as password fault so Amazon have got quite a helpful naming scheme so that you can you can get objects from s3 by the web so if I go to so I'll open up this I'll open this have a new browser just to make it clear that there's no

right no I don't want any of that right this is what happens when you're doing upgrade just before demonstration right okay ah I don't want to know about new things right okay so right so Amazon as I said they've got quite a helpful in a helpful sort of way of naming buckets I could have just clicked from the link there might have been easier but I wanted to show it like this so yes so it's HTTPS I must be secure and then yeah and then you put in s3 - then you put the region so it's like II West - one then you pour amazon.com AWS calm then you put the name of the bucket which is CPMs three

then you put the name of the object so this is password - file dot CSV so do that so I've just downloaded that file even though the bucket was not public so you can see here you know I've downloaded it this you know any of you that wants to download any of any of you who want to download this uh you know can download it so okay so how come yeah why did that happen any any ideas

pom-pom ACL exactly I knew there'd be you know at least one person in the room so access control this so you can have access control lists at the bucket level but you can also have separate permissions at the object level you can also have bucket policies which JSON based policies so if we have a look at so although at the bucket level there were no permissions if we then have a look we go back to the overview and then have a look at the password file dot CSV and then we look at permissions within that object then yeah you're absolutely right there's a different access control this for the object that is greater than the parent bucket and everybody is

allowed to read this so you might think well why would anybody ever configure anything like this well so Atwood app one of the organization's I was working with a DevOps person created a public s3 bucket so I said to him we don't you're not allowed to create a public s3 buckets you know make it private please but to say please so he did he made it private but when he made it private I think that all he did was just make it private manually and left the ACLS of the objects as they were so so this sort of thing can't really happen it is very easy to miss configure particularly when minoo particularly when you're doing

everything in code it's a little bit more obvious through the GUI the link oh so that's anyway so that's the second issue which is s3 buckets so now let's just go back to the PowerPoint yeah back to the PowerPoint and talk about next thing which is keys in get up so as an illustration I created to github repository so if I go yeah oh yeah I actually got it here so github repository whoops yeah okay so github repository so this is in my own organization github and I've created a repository just so that nobody thinks I'm doing this on purpose I called it insecure just to sort of illustrate you know how easy it is to how easy it is to

put keys in to get up so when I was at one of the organization's I was working with I was at work and fortunately I will have my email open and it was you know eleven o'clock as whatever and got an email from AWS saying somebody's pushed a secret key for this account into public github luckily saw it straightaway and immediately loved ones the console and deleted the key so within a minute or so of receiving the email luckily it was while I was at work looking at emails and so on and we then you know informed the developer who I think actually by that time and realized what he'd done the so that was fortunate

but it's often the case that people aren't that fortunate so for example I just get rid of some of these things

yeah for example DXE not to particularly pick on them because this can happen to anybody really for them it didn't happen during the week at 11 o'clock it happened on Friday afternoon unfortunately and somebody accidentally pushed a key brother said when they came back in on Monday morning there was a 64 thousand dollar bill from people who mined bitcoins using their account fortunately also for them well fortunately for them it was a development account not a production account so they said that there wasn't any customer data in there so that's easily done and I do have a confession to make it's very easy it's very easy to sort of say oh that's developers so stupid how

could he do that but I do have a confession to make which is that two weeks ago I did almost the same thing that was with some Azir credentials and fortunately wasn't a public github it was to a private repository you know within the organization in a private network luckily but say leave you've worked with code and get you can have variables files and then you can have a doc git ignore file that tells get what not to commit and what I did was to rename the variables file and I forgot because I was in a bit of a rush I forgot to update the game nore file accidentally push my zero credentials which were for my own account not a

company account and fortunately my co-workers didn't take advantage of it and my the hullender bitcoins I realized pretty soon and change the password but it really is very easy to do AWS have got a repository called AIDS over there - secrets which is some code that can actually stop you from doing from even committing anything with with the key with keys in which might be worth looking at if you if if you're working with AWS so that's one that's that's sort of the I guess the third issue I've also seen more slightly more subtle variants of this such as not an AWS secret key being on github but a read-only key to and organizations github and one of the organization's I

work with one of their developer one of their dev ops people had in their own private personal github repository had that they just were using for testing they'd put a read-only github key to the corporate github so anybody with that could then access private repositories in the corporate github and within those private repositories there was quite a lot of other keys which could be used to take over an account and do a lot of bad things so yeah it's not always yeah yeah it's not always quite as you know black and white as just the age of your secret key there's a lot of information that can end up and get home which isn't as

you probably know now being taken over by Microsoft right so that's the third issue so okay so what I'd like to talk about now and do maybe a couple more demos of some of the things that you can potentially use to help prevent this so first of all why did these security issues happen well typically the these sort of the organization's I was working with there are maybe not necessary thirty but twenty to forty DevOps people and cloud ops people I'm sure a lot of you have worked in an agile using agile methodology so typically we have two-week sprints and within a two-week sprint you have individual stories each developer or DevOps person picks up a

story which might be to create a new feature add a bit of security or whatever and then starts work on that will create their own branch potentially implement it it'll have to go through CRC deep pipeline testing usually but always and have to be reviewed by somebody else but you can see that where if you've got 30 to 40 people doing this every two weeks and you've got maybe each one might pick up two or three four five stories in a sprint you know that's a hundred or more different changes that are going on and any one of those could potentially include some of the errors that are sort of shown you that could lead to compromise of a production

account so that's sort of part of the background but then you know what could we actually do about it so as I mentioned at the beginning there's several different approaches there's technical approaches and then there's the sort of training cultural organizational type approaches and we need to use all of them you know you can't just rely on on any any one of those so on the technical side some of the things that we can do are to implement policies code which can ensure prevention so in Azir you can use as your policies which can do certain limit what actions you can take with AWS you can implement organizations and service control policies as well as iron

policies security tests in the CRC the pipeline security scans so they have more fun on the dissection side and then you can do some enforcement so if you detect something that's out of compliance then actually go and so let's do a have a little look at a couple of these tools just to give you an eye idea so so what I'm going to use first is okay so I'm going to use a tool so if there's anybody from NCC in the room yes so okay and and you're leaving how could you leave what I'm about to demonstrate your your tool oh well fine what I was gonna say that I'll still say the entity see if I'm very well - and

it's very good as them to open-source their Scout to tool so let's have a go at using it so what I'm going to do is type Scout - and this is all along github so Scout - then I'm going to do profile which this is just because I've got more than one AWS account set up this just tells the AWS command-line interface which account to connect to so I'm going to connect to the same account in which I gave the s3 bucket demo so the name of my profile is just the name as that account and then all close to it and then just because you don't want to be sitting here for ages and I know

lunch is coming and I've nearly finished so I'm just going to limit it to save time to one particular edge of your service rather than do a complete scan of everything so what it's now going to do is connect to AWS and have a look at everything in that account within the Identity and Access Management section so it's running a whole series of tests and I'm just over writing this because I've done it before so yes you'll see here that it's picked up under a dent in Access Management a total of well there's 12 resources 28 rules and 8 findings so let's have a look at some of those so you can see here that I've just got a good snapshot

of this particular AWS account red obviously means high-risk green means is well configured yellow is medium gray means it's not applicable or can't be checked so you know I'm very quickly identified that there's a whole series of issues that could be improved with the configuration of this this AWS account I mean that's not to say that you know as good as this tool is and it does similar test for s3 ec2 and and so on you can't assume that because everything is green on this tool everything's fine because there's certain things it won't pick up so for example this would have picked up the first issue the ion roll issue it wouldn't have picked up the s3

object ACL being public because it doesn't go down to object ACL so nothing's perfect but it is a very good start and it's largely based on the sensitive Center for Internet Security this benchmark for AWS on the repository side there are tools that we can use so I mentioned before we had I mentioned before AWS secrets can be used to stop you committing code with the edge of a secret key but sometimes it's too late what if you want to just find out what's there so there are tools that you can use so I'm going to the the I'm going to scan the repository which I showed you for the web GUI earlier so I'm just

going to do get Rob because this is get Rob also on open source and is us spelling and then the name of the repository so I'm scanning github and what it's doing is it's looking all the repositories and the other quite nice thing about this is it doesn't just look at the repository of scanning but if you've got contributors to to that repository that it goes and looks at their report their public repos as well so let's have a look at what we can see on there so 9 3 9th Street so if we just go to ok so you can see that it has found various things for example it's found an RSA private key there's lots

and lots of private keys in github and this is one that I put there but don't worry I'm not using it for a SSL website and also it did also find the AWS secret key and that is the activators if anybody's tries to use it feel free to have a go so yes so that you again this is not foolproof it won't find all credentials that are committed to repositories and it will find some false positives but it's a good starting point so just go about the presentation which is now nearly finished well close to being finished so we talked about some of the techniques we could use technically training is important so I've developed a training

course for QA which is GCHQ certified but equally there's so much training out there I recently used a cloud guru for some address security training which was help me so that was that was very good and equally things like lunch and learns so when I was working at some of the organizations that I was working with I did sort of 10-minute WebEx sort of 10-minute sessions at the end of stand-ups on a particular security aspect like some of the ones I've just shown you to all the DevOps that was quite useful so there's a lot that you can do culture it's essential that senior management support security and that prioritization that security features actually get prioritized

because quite often the same people will be developing these as doing creating new features or you know providing performance improvements so yeah so and doing other things like gdpr compliance so it's a born to have a strong security lead that can fight for that priority and senior management support and finally organization separate security departments don't work with agile and DevOps and cloud I can confidently say that from experience what happens is that these guys just get in the way of these guys and they try and stop things from happening and they don't and it's just not effective at all the what is effective is embedding security specialists or training DevOps or developers to become security

champions to take a lead in the team so for example you know I every morning not today but other mornings you know I participate in the stand out I'll report on my security stories and I'll find out all the other things that are going on within the DevOps team so if there's something I have a concern about I will go and talk to the person we all sit together and we're all part of the same team that works much more effectively than than that so as it's lunchtime I'm won't take any questions with everybody here because I know you all want to go to lunch but I'm around the rest of the afternoon if anybody does want to ask questions so

thank you very much [Applause]