Red Teaming AWS: Practice What You Preach

it's also i think we are about time so let's kick it off um first of all welcome youth um i'm super excited to see you know your presentation that will be good spray or something so i can't wait here to learn more you know about it and oh the floor is yours good luck thank you very much thanks guys all right hopefully i know we're uh towards the tail end of a two-day conference i'll try and keep this upbeat as much as you can in a remote sense it's a shame because i absolutely love barcelona as a city yeah unfortunately we had to do this full remote as we have for the last couple of years right

it's a shame and don't worry if you are feeling a bit tired the bit that everyone remembers from this talk comes in the first 20 seconds and i'll get to that now so this is amazing a word amazing word in japanese called tsujigiri and so far no one's ever known what this meant when i first said it but no one ever forgets it when i tell them and normally i ask the crowd there isn't one but this is fine but the definition for it is is trying out a new samurai sword on a random passerby kind of a weirdly specific term and is based on actual history and that's what samurai actually did and that's the kind of ethos and spirit

we took into kind of the story i'm gonna go into today unfortunately i have to do an obligatory about me slide because my ego won't allow me to not so who am i i'm josh armitage i'm the adress practice lead at cantino cantino ra global digital transformation consultancy we work with highly regular enterprises we're talking the the biggest of the big why you should listen to me about security i lived in a prison colony for 13 years most people know that as australia but it still works right and also i am um the author of the cloud native security cookbook with o'reilly that's coming out in about may the next year so i normally don't have any

weekends anymore because i'm busy writing that what this talk really is about is shift left learning and we always talk about shift left and security and bringing the tools so you find the problems quicker really this is about what happens when you try and bring the learning and the experience to happen quicker right you know everyone's seen this kind of curve before this graph is nothing new but you know over time cost doesn't increase linearly it increases exponentially and what do we do if so if you can bring the learning earlier can we get that same kind of cost benefit the things get a lot better if we bring it forward now i'm just going to go through quickly

a few different mental models that i like to use in this space so hopefully when i talk about other things we're all kind of singing from the same nim sheet so the first is the knowledge buckets which came from a i normally look this up but i can't remember now i think it was a secretary of defense in the us effectively and he said there's three kinds of knowledge you've got your known knowns you've got your known unknowns and your unknown unknowns so effectively the things you know that you know the things that you know you don't know and the things you don't know you don't know yet and normally the unknown unknown is the really interesting piece

of almost doing anything because it's all the stuff you didn't even know you didn't know right so we're going to go through what for this what the known knowns were so the story i'm going to talk about today is what happened when uh at the consultancy i was at in the previous life in the pre-pandemic world australia where we took the case of what happens if we attack our own aws environment but don't tell the majority of the company that's what we're doing what happens how do they react all that kind of stuff right and this was instigated by the ceo i've had the question before like did people know well the ceo ceo and cto told me to do it so i

took it as i can do this and this is fine i'm not going to get pulled up the next day right and we had a bunch of known knowns or things we believe that we knew that were true one thing that was really important to us was security is everyone's responsibility it's not your security functions responsibility it's not the engineering it's everyone everyone has to be part of security everyone has to be as good as they can be within reasonable limits for security to be effective the second thing that we had was we had a defined instant response process is pretty much every business especially technology business under the sun has right it's in the paging confluence

where it was in this case people read it and they sign off on it and maybe they're forced to read it every six months or something but do people remember it when the pressure is on does that really happen this now seems like a luxury in modern times back back in 2019 when we first did this we were able to all be together and co-locate we were a consultancy naturally we were split across a variety of different places across the city where we were for this we didn't want to add the extra complexity of everyone being desperately and also people being on client side this was something we wanted to invest in as a company

we had about 30 people at the time we pulled everyone off clients for that day they didn't know this was happening but the company effectively said we're going to make zero money this day because this is important to us and i'll kind of i'll go on about that a bit because this is part of the power of this another thing that we thought we knew we thought we know what we thought to a degree we knew what people would do we thought we knew who would step up to a degree we thought we knew how they would try and react so naturally being the red team and being there and doing os and all that kind of stuff we

started to put safeguards and traps and stuff in place before the day to catch people when they went down the path we thought they would and i'll go into those a little bit as we go along so then we had a bunch of known unknowns things we wanted we knew we wanted to figure out from this one of the core things when doing this is how do you measure performance now when i talk about measuring performance i'm not talking about using this as a stick to hit people like you didn't do good there was no good bad or anything in terms of this this was just so we can measure where we are and then when we do it again we can see did

we get better did we get worse did we stay the same it's not about it's about this continuous learning approach to things not about hitting people for being bad right it's very much psychological safety and that kind of ethos and that side of culture so we came up with a few metrics the first was time to identify from us starting the red team activity seeing that the other everyone else in the company didn't know it was going to happen how long did it take them to figure out that we were going to do this and that we were actively doing something time to contain how long until they'd actually contained me and the other engineer that were the red team how long

till they actually took away all their access and got it under control to the point where we couldn't cause any more mischief going forward and the third was the percentage of intrusion detected of everything we did once they contained it how much stuff of what we did did they even find another thing we wanted to understand was who will take responsibility so as i said this was instigated by the c-suite they asked us to do this and me and the other engineer were two of the most senior kind of engineers in the company so you took effect of your entire leadership structure out so as much as everyone was on site we actually went next door to a coffee shop

to actually do the red teaming from so we weren't there so it was a really interesting concept of what happens when you take away the people that everyone normally looks to in a crisis or something going wrong what happens who's gonna step up into that kind of leadership vacuum and take control another thing that we wanted to understand was processes break generally when you are three times the size of what you were when you put the process in now when i joined the company we were 10 people as i said we're about 30 at this time we kind of had an idea that the processes we had probably weren't going to work anymore because we'd grown that

much 20 people doesn't sound like a lot but going from 10 to 30 is a massive difference and the last one was what lenses do people possess now in that third in those 30 people we've had a lot of developers we'd hire a few operations people were a devops consultancy we hadn't had that many security people and this was a case of do the engineers we've hired and the operations people will be hired can they look at problems through a security lens do they have that ability and again in a way not it's a pro it's not a problem if they don't because then we know the gap and we can fix the gap but are people be able to look at things

the right way and normally when i talk at these conferences mostly security people right and then they always complain about engineers they don't see the things the same way that they do it's perfectly legitimate and it is a real thing but this was we wanted to find if we had any like security people deep down inside within our company already that we could look to build on or exactly where we were obviously with the unknown unknowns we didn't know going into it what we were going to find we knew we'd find things we didn't know that we knew we're going to find as we went along and really what i like to bring this back to is the same concept behind chaos

engineering and since i did this someone else has coined this process the security chaos engineering which i found quite interesting because i saw the o'reilly book on it i'm like oh i miss my timing on this but that's fine and now you think it's interesting that chaos engineering in the operational space with gremlin and chaos monkey and the simian army and all that kind of stuff is huge now right like it's a massive thing and i think from you know this is kind of like pen testing but it's also like assume your pen testing is successful as in you've breached and you've got into your cloud environment what actually happens then so assume that you your perimeter has

holes because any perimeter of any size really does you know day zero events are legitimate things what happens next if someone gets into your cloud environment and i think this chaos engineering security cast engineering is going to become more of a thing in the future if you're looking to do this yourself um set up some rules of engagement we did thankfully we had a maximum bill in terms of cloud bill we were able to incur we couldn't just go stand up the biggest ec2 instances that aws could provide us and bankrupt the company in a day we had to focus on keeping it under control with the amount of money that we were spending all that kind of stuff

there were certain things we weren't allowed to do like we couldn't go and kill identity effectively and make it so the blue team couldn't do anything now legitimately someone could have done you can do that to a very high degree in cloud but that would kind of stop the fun and the learning it was about not about making the blue team feel awful about themselves because they couldn't do anything but get that right level of challenge and right level of competition that they always felt they were close and we just keep moving that little bit forward not kind of make them all depressed after an hour of finding out they couldn't really do anything right

so with that that's enough of me practicing on about all mental models and all that kind of stuff so i'll actually move into kind of a replay of the day so 10 o'clock in the morning you've got to let people have caffeine before you do this otherwise that's really unfair so let people have the coffee and we started so one of the things we tried to do and i think this was really important is roy always looking to how do you channel learning through people you always run the risk in these kind of scenarios but all the most senior people will go okay we're going to solve this they go ahead and heads down hands on keyboard and all

the junior people kind of sit back and go well i'm just going to watch them do it right so one of the things we did is prior to kicking this off one of the first things we did is we killed every senior person's access at the company to adress we left a lot of the junior peoples around but we killed all the senior peoples with the intent that then the senior would have to work through the junior to do what they wanted to do so very much like a highly driven approach for learning like we were big proponents of pair programming it was very similar to that kind of model but making sure that the people who had the most to

learn were doing the most work um three minutes after we started we tripped the wire now this was a serverless consultancy first and foremost we had an alarm that would say if someone set up an ec2 machine an alarm would go off so we knew we knew how the trip wires and how to set off some alerts to see if people would do that we didn't want to run for two hours before anyone realized that something was going on right also at the same time we sent out a pre-crafted phishing email from that looked very much like the compliance tool that we used that triggered those alerts as well so we did fish to see if people would click it i'm

sorry to say people did people always do but it was still you know we wanted to see the percentage right it just always happens so nine minutes after we started someone noticed um a guy called paul put his hand up went okay i think something's going on i'm going to investigate he very quickly figures out if he was one of the most senior people in the room his access is already gone so he asked for someone to come and help him uh zynab went over to help him we in we instantly blatted her credentials before she could do anything as well now i've already said we were in another room how did we know what was happening in

the room and we actually had a double agent on the inside on the blue team side to feed his information feed us how everything was going in the room partly sure we could stay one step ahead and partly who knew if we needed to pull back a little bit because the stress level was running too high or anything else you know it really is you don't want to make you want to turn this into a really positive experience for everyone not a really negative one three minutes after that it was called in the company slight channel this is not a drill something's going on everyone needs to stop what they're doing and come and help

five minutes after that the ceo was called and i had great fun watching him sipping a coffee look at this phone and put it face down on the table and just go well i'm not answering the interesting part was that is step one of the instant response procedure was called the ceo that's how long it took 17 minutes one of the interesting things that came out of this and one thing we really didn't know was how are people going to structure themselves when this happened right you've put people under a lot of pressure a lot of stress and gone self-organized and i think what actually happened is probably going to be pretty common if you do this

on the first time is you ended up with something like this there was one person paul at the top the initial person to realize everyone was feeding information to him and naturally the next thing that happened was this his head metaphorically set on fire there was too much information too many people trying to talk to him everyone was trying to feed him back information and ask what to do at the same time and he just went oh my god i can't handle like something needs to change so at 10 28 they went okay let's stop and regroup let's figure out where we are what we're trying to do let's get a bit of thought behind this as opposed to

everyone running around in circles so at 10 30 they did an access poll so who's still got access who hasn't that defines you know how power what they can do how much they can act on who they need to kind of be sitting at keyboards and then when other people can think about things a little bit more and then you know kind of work through them and that kind of stuff what was interesting with this is someone said he didn't have access when he still did as well which i found quite amusing on the day that someone didn't feel confident enough to do that now we didn't call it out like publicly but in person you know in

person i went to the guy after when okay cool like it's fine that you did but how come i just want to understand why you weren't comfortable kind of pointing your hand up as having access from there with that organic structure they split into design ab the person who initially helped paul in the first place became like a communication pathway and paul was able to kind of extricate himself to a more full leadership role so all the information was coming into xynab and she was unable to filter and give paul the pertinent staff not absolutely everything going on and he could get that extra space to think about things and once they did that things started to

move a lot faster on their end this would be a lot more controlled a lot more getting a lot more done than they were before when everyone's running in different directions uh uh spy the double agent i talked about before people were very curious why he wasn't being very helpful they realized what was up because at this point we had told them it was a drill um to treat it realistically and also it was now a competition between me and will on the red team and the blue team so if anything this maybe made them work harder but we wanted the stress levels in the adrenaline just to drop off a little bit so he did tell them it was a

drill but treat as if it's real and you know naturally they keep the double agent out and he came and just sat next to us and grabbed a coffee and had a lot of fun watching seven minutes out after that i managed to find github credentials and i broke out of edus because someone had missed stored their greater credentials badly cool all right now you know from this initial red team aws point we've actually broken out into other systems and this is kind of the problem right like secrets and credentials get everywhere people need to know how to store them properly and this always brings me back to one of the core components of a res which is

kms or kms on azure or kms on gcp kms on whatever cloud you want a fundamental understanding of the key management service is absolutely critical for absolutely everyone that does anything in that cloud it's something that a lot of people shy away from and they do badly because it works even if you do it badly like you're still able to kind of move forward on your future delivery but from a security perspective you just end up leaving stuff kind of open to the world normally it's i'm in the account i can get to anything kms or no kms because people haven't said the keys properly so if you have one kind of target area in any of

the clouds to go after to upskill from a security perspective im and kms the two kind of constants across all the clouds though that's where you want to target okay 10.50 also on the western front they eventually realized that talking on the public slack which had the red team in it was probably not effective for them being able to actually counter actors so naturally they went to google hangouts and made we couldn't see what they were talking about anymore they got rid of the double agent and they went off slack we didn't know what they were doing anymore naturally at that point we went and threw the cto and co back in and go hey

guys you need to go back in and help them but also rebuild that communication line again to get kind of the right levels of communication going back and forth three minutes after that they thought they had us contained uh they didn't it took them almost 20 minutes to realize they didn't actually have us contained and will and i were getting a little bit more audacious in what we were building five minutes after that we sent them all a photo of us doing things which i can't put on public slides with our fingers up to them just to kind of have some fun with it all and eight minutes after that they managed to containers 22 minutes after that

will and i stepped back into the room and got flipped off by everyone in the room um which is quite a feeling just walk into a room of people you've worked with in the drenches with for a couple years and just have them turn around and go i hate you now you know it's quite fun if you haven't done it i do highly recommend it and then ten minutes after that we moved from contain like we had those contains now it was remediation and we had set a boundary on time if it got to one o'clock and we hadn't been contained we were just going to stop and move to remediation anyway but they did manage

to contain this in time but still always having those kind of lines in the sand drawn so you don't take it too far and because part of it is getting the learning bit and everything else you don't learn from experience you learn by reflecting on experience so it's making sure we had enough time to reflect on everything going on a big part of the ethos of that company was kaizen change food continuous learning that's continuous improvement that's really how we tried to live and move forward and that was really important for us with this so let's kind of step back to those measurements we talked about before time to identify eagle people with good memories right

remember was 12 minutes the time to contain was one hour and 28 minutes and the percentage of intrusion detected ended up being about 66 percent again i have no idea whether these numbers are good or not i have no idea if you did this at 10 different consultants whether you get better or worse than that or whatever it's really not the point the point is about going internally and looking how you can improve it's not about measuring yourself to everyone else in the industry it's about how can we be better tomorrow than we are today one of the questions i got which i found really interesting is someone asked if it was realistic what we did

effectively all we did we didn't do anything more than a um someone who got fired and still had access or was anti you know it was just one set of address credentials was enough to do all of this and we set everything up and did all that kind of stuff it was just one compromise of one identity was enough to do all of this it was interesting because we tried to keep it realistic and people still asked us whether it was it's like no no no no it really was that was a key thing we're trying to do not do something like completely out there that would never happen in reality no no this could very

easily happen tomorrow like even with mfa and everything else and still one one employee gone rogue and this can happen one of the things i always think about is backups never fail restores do like just because you have a process or you have stuff defined you need to test these things you need to make sure that you actually go through the motion and that was with the instant response process people need to go through it to remember it you can't just expect them to read it and have it sink in right doesn't work i think we all know it doesn't work if we still do it anyway um one that i i swear i am coming close to

the end there's only a couple of minutes left and i will try to end on time might go one minute over one thing that we noticed from this one big learning from me was going back to john boyd and the ooda loop which i think a lot of people in security have heard about outside security a little bit left so john boyd was a fighter pilot known as 42nd boyd because he could beat anyone in a dogfight in 40 seconds effectively and he had it all based around this particular ooda loop which he called which he coined the term for and the idea is you can observe orient decide and act and if you can go through

this loop faster than your opponent or your competition you should be able to out maneuver them and win effectively where this kind of ties back into this is the fundamental problem the blue team had or in my opinion what the blue team had was it took them so long to observe and figure out what was going on they could never really hope to catch the red team in what we were doing there was a massive gap in observability in terms of being able to see the estate see what was happening the change that was happening and everything else and really the end of the day we it was a lack of tooling they didn't have the

right tools on the ground to be able to interact this and this was fantastic learning for us okay we need to go investigate something we need to go and find something to move forward we identify the gap again not a problem that there's a gap it's a problem if we identify the gap and we don't fix it you know we captured all the improvements everything any idea that anyone came through throughout the entire day were captured they were put on a giant jira backlog and then there's a consultancy anyone that ended up on bench not on an engagement they just pulled away at this backlog over time and it was just like a consistent rite of passage you join the

company here's a security backlog work against this or if you're between engagements cool like here's some security tasks to be getting on with like it became a beginning of a tradition from this we ran both internal and public ctfs this kind of story is still talked about at the company with frightening regularity every time we do a team day and everyone's pulled together on site no when i was still at the company no one trusted me anymore and everyone would look at me suspiciously from that day on every time we're all off site because everyone was always like maybe they'll do it again and we you know they will do it again and i've done other

companies now as well and it's just really effective i'll just finish off very quickly on why red team why investor time and the money and everything else why go get the buy-in from the c-suite to go and do this and fundamentally i come back to chess on this which may be a little bit boring to some people but i really like this concept the difference between a novice and a master in chess is a novice sees all the pieces of individual pieces and that's the way they look at the board a master looks at the board and sees groupings and patterns and things they've seen before they have that experience internalized they see a very

different board to a novice and really when the trips are down when a security incident happens would you rather have masters alongside you or novices and the choice is yours with this do this you may have some masters don't do this you're going to be surrounded by well intentions but novices and you're going to have a lot more problems because of that that's that's it guys thank you that was an incredible presentation i was kind of you know picturing on myself being ideal you entering that room so that moment you know like that it might be going on my like gravestone when i die it's just like some like artist rendition of that because it was it was truly like yeah an

amazing time so i don't think there are any any questions you know for you but i got one if you if you like me so the if you could pick up you know just a single key takeaway or learning you know from your exercise that you could recommend the companies you know to invest the time and effort you know what would be you know the key takeaway i think i think for me one of the key takeaways is um culture is made through action not words if you invest in these kind of activities and do these kind of things you make you like kind of build into the culture of the company that security is

really important it's something everyone has to do and it's something we actively invest in it's not a bolt-on at the end but these kind of things like the average security company of everyone in that company after six months after this was three to five times higher than it was before we started this because we made it obvious it was important to the company and that kind of stuff and it became part of the culture so i think that's the thing like even if you're not in the cloud or whatever these kind of scenarios they just make it important and make people realize how important it is follow do you have any anything yeah i i wanted to ask like you did this

this red team like being you part of the team have you tried like with like external one and being like on the other side in the sense like you have like probably a lot of internal insight that another red team don't don't have like you do you compare the experience yeah no normally we normally do it with the ideas that any any act will have done their research and done their own and that kind of stuff we expect the red team to be able to target very specifically to whatever you've got internally and that kind of stuff i think it works better that way um that you do want that kind of intelligence of what's around there

because it allows you to be smarter about what you're doing and also if you're talking state actors or you know some of the real threats that exist for certain businesses they are going to do that it's not just script kiddies downloading stuff off the dart web and running it really i mean those exist as well right but when you're thinking about this you should be playing under the expectation that they know just as much as you do with your architecture um so no i've i've been fortunate to always be red team centric in this i think i do okay in the blue team side but normally i'm i'm i think i have the mindset and people

see that like i could i don't want to say cruel but like i could i take quite a bit of pleasure out of doing this on the red team side so they're not like i will get you to do it because you'll do it right but you know i think and you know running the red team and getting that kind of gap i talked about but you're not pushing too hard and you let people catch up and feel like they're you know they're making progress and all that kind of stuff that takes a bit of nuance and a bit of experience to try and get right um because sometimes you just get people to go way too far and or

you know not far enough and everything else and just constantly keeping that competition level there's important as well yeah so you have like actual you know the people so you can better measure this level of pressure i guess yeah yeah yeah yeah the more you know the better i think the better job you can you can do right and yeah you want to know people you want to make sure you're not pushing people too hard and depending on the culture of the company and stuff like that the amount you can push and everything else kind of changes and all that kind of stuff the more you know the better an experience because again you want it to be positive for everyone

involved um because that's how it becomes something that kind of takes on a life of its own if people have negative cons people have negative experiences of it then it starts to die quite quickly yeah yes so we really enjoyed your presentation i hope to see you face to face in barcelona in future editions uh so i'll keep an eye on your book you know once it gets out um yeah all the best you know thanks for for your presentation and your timing in the in the conference nice thank you guys we're really honored to be here