Stacking a Blue Team with Red Teamers

speaker is David Greene and I'll let you introduce your talk and just take off here right here me good cool all right so type of a talk is in Gloria Security I don't feel like reading to you so just keep going

one day

okay so Who am I my name is David Greene I worked at Riot Games security analyst before this I saved red team at rapid7 and then for that I did just different information system stuff for the Marine Corps this talk is more about how we built a blue team with zero blue team members capabilities that we thought required to catch us make life a bit harder and kind of warm a salute to blue team because red team just kind of got to get right once and then go from there and it feels like every day taking Alice blue team this talk is not technical so there will be no code examples I will not release anything on github

afterwards it's not the only way to do things just the way we are doing things but red team gets to share a lot you know next time you see an exploit it goes up on github everybody else gets it chances are you're gonna see that in a while very soon we don't have an easy way to share so just trying to let everybody know what we're doing and hopefully it can help you and we can learn I can learn from you guys afterwards okay so it didn't we're gonna talk a bit about our company culture which makes some standard things like which would consider best practices not not applicable when it took to build our

team the capabilities were implemented lessons learned and where we see the future going in the next year so some context a publisher of league of legends so we're very focused on our customer which in this case is our player we want to hyper serve then we want to make sure that every little complaint they have we here we may not agree with your complaints where you we want you to feel hurt and we want to make sure that we are catering to you and we are your first and foremost everybody is a gamer in some way shape or form they enough maybe not be video games feel like there's a person on our team who just plays poker all the time

on the weekend common so our mission is we aspire to be the most player focused game company in the world that which can come with a bunch of challenges because developers you know we sit you down and go through our in dog classes it's like a week-long you come out of that you're like alright I'm gonna give players everything they want I'm gonna ship this the codes gonna be awesome at least in my opinion because every code I write is it's amazing and everybody else's code it's terrible so this ship and they ship a lot of things right so as of four years ago and it's funny growth we have more million active players in the game 27

million daily and seven a half million concurrently so why would you attack us well just like in another major tech company out there we want to ship fast everyone wants to be agile and when you're doing that a lot of time security just comes at the cost of speed you need to ship your core game thesis before anyone else does game development takes years seven years eight years is not abnormal to take a game from nothing and ship it so and if you're shipping a shooter or you're shipping pretty much anything unless you're inventing a whole new genre it's a very very red ocean out there gamers are very opinionated and fickle like they have only so much free time to

play and you know why if call duty isn't working for you then you go play battle you can play overwatch so we have to be very sensitive that large attack service right if you're especially if you're a PC game maker like we need a support back wall we no longer do but for a long time we need to support back to Windows XP so we have to have all these different test environments around kind of lock those down to make sure if you're able to pop us you can basically test rip your perfect test bed for excellence and not only that we're going to be working in basically every major web technology and you'll see so you'll see

anything from massive and ephemeral container environments all the way down to the oldest operating systems unpatched it's if you're a you're a non-us government entity and you want a place to play it's a good place and then large scale like for any game company if you're blizzard your us your about if you get your source code popped and your your distribution client because this case say for example steam not saying this happened but if your steam and you get popped and all sudden you just you know put a listener into 100 plus million machines across the world if you are not a nation state government I can think of several that would pay a lot of

money for that right off the bat so while we do work in games in the disclaim like the gravity of with we fail horribly is it's really really bad it would kill kill any company so InfoSec team break down this is generally the areas we work in we float pretty heavily the red one security operations is what this talks about there are other talks about their writers out there specifically around popular security that if you're interested at their although all right so building setups so the way it used to be was what I call 10th amendment security if you weren't if it wasn't this wasn't guaranteed to be platform sec it wasn't something right now you get this for sure it fell

under us and so because of that we got overloaded and then 2013 and 2015 we had a couple fairly bad breaches 2013's will know with these within games because we have our foreign password database extracted albums and there was a lot of credits lost to that 2015 that one was pretty rough and we were basically like alright we've got to change so too after 2015 the only thing we really learned was that we were bad we didn't find out about it someone else came to us like manuals like hey we're investigating something else guys might want to look here and maybe here and also over here great so we wanted to change that and the only thing that we had essentially

was a really strong robust logging PI point and that ends up becoming the catalyst for everything they need to know so we did a self-assessment like what should we be doing are we currently delivering any value how do we get there and can we show the business value tracing this book earlier but we have to make sure that everybody knows what we're doing this we don't want to be that security team where it's like you know it's a lot easier the core like the quickest way to make sure a feature doesn't ship is to invite security so we don't want that so we came up these are the five things that we think we should

be doing and are our stakeholders stakeholders being the CISO and then his fossils are aligned on this one if anything we're working on does it align with these it gets killed off pretty quickly Tracy makes sure it happens even though it's somebody it's that project never happened so so we're we delivering any value so this is just a screen cap of like maybe 15 minutes Obama so yeah we have a shitload of logs we have a d-plus petabytes of data across the company so we have a lot of data but everything else need to get burned on the ground I think we did the whole process like we're so manual you had to like oh mac

alert we're from Symantec how are you gonna fix that you can't kick off an automatic remote scan you can't if it is actually compromised how are you gonna get all the boxes because all these really basic things we were missing and I was not told we were missing those things when I joined the company so that was a fun thing to find out so how do we get there my manager is like all right well we have three people who claim to be blue team and everything and they were overloaded so what we really want is people who understand what an attacks gonna do and so the only real way to get that I mean there are some people blue team who

gets it but it's really easy to just go out and find like security consultants oftens the guys people especially now the rise that puts a lot of people who leave government various branches and know what they're doing and not just American government so we want to leverage their mindset right that's actually how they got me in the ring and I'm not bitter at all there's like how many reports of you deliver how many companies have you gone to it like you pull up a report and last year you do the same thing get in this sucks I would like to fix this so they get to you by saying you go you can fix this

and you get to build a network that's gonna make it painful for yourself so if you or maybe a little vindictive and you want to cause other people paint this is a great way to do it and then 3-woods don't hire Turks like in our community this is a problem like there are people who want for whatever reason reward Rockstar status and people become can be really really difficult to work with but at our company like you don't get anything done without building relationships it's all about who's going to help you because I need something done somebody else's work has to be deprioritized and why is my time or what I need more valuable than shipping for a

player so if you have somebody who causes a lot of friction even though they might be the best person in the world of code we're never going to be able to work with them because they well we can't just chew in a silo by yourself every day okay so we're actually very fortunate we on our engineering blog we have more in depth information like this posted but for developers and security falls on our engineering department we have a very defined set of we call masteries and we give you the belts because we're nerds so you could go obviously from white to black and this is just four of the six major things but it gives us a great framework

to say alright I would like to have a mid-level security person and this is where they would hit like they would need to be an orange or red and a couple of these things like two tops and then everywhere else they would want them to be like a yellow we have this idea that we want a t-shaped engineer so we want you to be really really broad but then specialize in one or two places so it's we do this every year you look down you self assess and then your manage your self assessments and if you mean they're like just like come on white bells across the work okay so and this is a bit harder to see but I will release

this presentation how afterwards to look in case anybody's interested but in addition to that we say these are the skills that we want all of our security analysts to have we want you to have a very base skill in all these and then of course anywhere else we want you to may be highly index in a couple things that are tend to be unique towards ours and I just really enjoyed the last talk was that we have a lock threat up in a station using human because surprise surprise you have some of them in concurrent players they have a bad game it's kind of bad day chances are player support is going to get death threats

I'm gonna come to LAX and shoot you up we are going to blow up Madison Square Garden why they're the only list goes on like people get very very upset about our games I'm one of them it's not hard I still type in that's how you don't get bad right so the other thing that we're very big jobs we have very short Job Description so we're hiring like this is for a platform security engineer these are this is just barracks five bullets this is what we well most the things that that got me to come with her I was you see a job description and it's like someone's trying to like play pokemon and catch

all the different descriptions out there like oh you want someone even is networking you know tcp/ip what about me how about DNS okay we get it you also he knows network being cool just say that so we kind of want everything to be brief so we said we want to break into three teams because we feel like security operations covers red blue and everything that supports it so we have threat attack detection response in the tool so threatened attack are the people that whenever it says do you have a red team a riot the answer is yes they work inside threat attack but people will float implode everybody are to you should be able to

float across these they have their own leads but you might do six months in detection response and then you know you may not like it or get burned out so it's the dealer to do tooling or he sent over to you threat attacks everybody stays fairly well-rounded but threat attack will do threat intelligence a lot of ours is custom source we'll talk about that a bit later we do read to you as a service so there are some products that we use where we have a product catalog and say alright these X codes we know work it leaves these forensic artifacts and we have this scenario so we can go to any environment and get

stakeholder by handling for example I can go to environment there's a lot of different build environments to the different types of software we ship all right well what's the risk over here run scenario and say alright see actually we do have all your credentials right here this would you turn get us into github so you could and you didn't configure everything right here so yeah maybe you could just go inject fairly quickly here if you were to get popped and these also people to handle player threats so whenever lose the gum threat her school shooting or anything else we handle those and it's also a part of the team that does an hour analysis and

research so that's that's like a reward you're like I did a really good job it's been three weeks just reversing something detection response is kind of what you would expect it is it's forensic to scale but globally we're in as you want to say at least 16 countries with 20-plus offices so we would like to be able to do over the wire forensics as much as possible because nobody's going to be getting a terabyte of a full disk capture anymore especially and then shipping across getting it out of Turkey getting out of China through the Great Firewall not easy and the tooling side like mostly they helped us build tools to make our life better or we hand off

tools to them like I made the super cool script and then they come back and tell you like ah not so fast first thing so infrastructure is code they want everything so you'll pass it to them and they'll work on making sure it stands up in Hanceville or a packer and terraform so we can tear down whatever you're running and bring it back up make sure that we have really really strong cyber liability anything that becomes critical or a workflow like me also we want to know when it fails so ends up getting paged out we've had a handful of things words like oh cool logging pipeline went down for four hours nobody knows not the most fun time

to explain that engineering like why didn't it work so how did we get there or how do we get there security operations is essentially we are the customer of two things corporate security and platform security platform security being people who are primarily interested in the games and their cloud footprint and corporate security is everything else so and then our customers are anybody who has been like alright we want to have forensics at scale we want to have all we think whatever we have needs to be protected more than just what's coming from corporate security so please help us it's important to establish this because you know you can get overloaded because most people think all right my job in

sprint six are my job is instant response and I'm responsible for the whole company but if your tooling suite and everything else isn't there yeah you might technically be but as long as everybody else can acknowledge it's gonna be in a degraded state you will hopefully come out better because otherwise you should have many many sleepless nights worried about things that are way beyond your control so controlling your scope that way is it's a lifesaver honestly and it allows you to have nice metrics check in the box for management we awarded this week a new environment today we're doing something so one of the things we have to have is minimal minimal impact to shipping anything any

features so our endpoint solutions generally we don't want you to increase anything that's going to increase the bill types or it has to be negligible because we've run a game as a service we want to ship a major patch every two or three weeks there's a ton of bills that go involved with this so if you if we increase bill climb by 10x by 15 exits and noticeable we don't want you we don't want to delete na malicious files we had a lot of problems with that we're like you know hey we're a software company we make new software I realize you haven't seen this software in awhile before please stop deleting it and then we need

to do is we want software that just reports on all of our processes just be a snitch I can go back as long as I save the data in the login pathway I can run an investigation six months back no problem as long as I have it contrasted that the logs and the snitching is done properly so we're good metrics for us we actually have no idea so if you anybody has a good idea what what this looks like please this is actually I pulled up earlier this week you can see that we ignored 27,000 alerts fantastic so implementing capability asleep how we did it was everyone gets upset when security is like all right this is the this we want

to protect for this audit finding or we think this is the best way to do it and this is risky because everybody's gonna push back we happen to be the company where most people are real subject to magic matter experts in the field and if you come to them with some like well this is the risk and you know they know it better than you you just lost all of your credibility with them so you have to be very very careful with that so what we did is we came up with a tent where we think the ten things that would kill the company the most like if this happened tomorrow we have to shut our

doors were we're just on and then the c-suite came back I said well these are the 10 things we think of so and there's some haggling back and forth and they sacrifice their system to the gods and we came up with the strategy or a list of 10 environments that are the most important things we want to protect and their stack rank though and then after that it was build relationships with these teams motivations with the team leads the product owners everybody's going to be difficult so you got to figure out how to handle them so that's kind of a little about humans so if you like the human talk the last one figuring out how to deal with people

that's a trick so we said these are the kind a lot of where you slide I'm really sorry for that so this is what we think kind of like a five-star experience would look like if you're working in a sock now we want to make sure that anything that we think is Alyssa or anything that comes in through threat Intel just gets automated get to birth through it gets to now analyze and they kicked out to our pipeline so we can push it across the rope anyway this is just mostly for atomic because dealing with atomic is you know its base level one I think it's not where you want to end but it's a

good place to start but everything should go on a single pane of glass so I don't want to be in Palo Alto looking for things and I don't want to be looking carbon black or anything else like that I want whatever they have turned into like whatever the law of karma has brought in normalize and and kicked out to what I want to see because every time somebody has to switch context you're losing time they're losing lose focus and it makes it much harder to hand off a ticket so threshold reaching a threshold of severity that point is a little bit more tricky we still have quite solved how we want to handle threshold but we definitely

figure out how to suppress rights if anybody has a good idea on temporal that's what we're trying to solve right now so if you see an alert and it happens here then we want to look look five minutes after and five minutes before four for example recon commands there's a concept so like oh yeah whenever I land on a box I want to know who I am or run Who am I Oh who's running away my across my environment Oh 30,000 hits also this isn't work remediation work needs to be automated this sucks because people are going to download pups into lists and somebody's got to clean them up so you can do an automated way you're gonna

reduce your noise otherwise just going to set their trigger over and over again to your 27,000 parts that you've ignored for the previous week and then everything should be exposed via an API we're being on building custom we're not very good at integrating vendor products the way they they think we should so expose everything via API and then so we can chain our home to link together so how we work these are the main three tenants like we deal with alerts a lot everyone is in deal with alerts a lot but alert is unplanned work you doing I'm playing work you're just gonna crush everything else going on so we do like a lot of other places we want called a

spotter sniper or primary secondary but we want to make sure that there's one person who's primarily fielding all the things that suck and in turn they can help automate it away the next week project work if you're not on primary then you're gonna be available for project or gonna be attached to something there's not going to be a lot of downtime but then flexibility like you you might have every Friday somebody come and say oh I did this thing oh I posted this to github this AWS key or something else and then you've got to remediations everybody's got to be flexible you need to be at least as flexible and your company's deployment cycle is so where we play we improve

capabilities we we reduce log burden like if you're all locking every packet that comes through like we do we're doing you can get a multi-million dollar bill - fantastic we tie capabilities together via mostly via an API chaining and then we know we advise people say like you know we want to do this how can you know when they're gonna come to you or the dream is for them to come to you and say we want to do this how do we do this securely first we may not know how you do it securely right off the bat because somebody will or we will know what we'd respond that apartment so for us we always like to be moved in

early and give advice on how we can make life easier and better for them what we don't do is we don't implement brand new capabilities so what we do have a full packet capture and we request it for a lot of time like we're at the mercy of somebody else to do that rather than that and usually a corporate security and they did a fantastic job I mean when you're when you're living in alerts every day is painful especially a couple in the red state right you you get in quickly you can pivot quickly you just the speed that you can move when you're on offense as compared to defense is so much higher and so you feel a lot of

pain there but you have to be very careful you need to have other teams ownership other teams by them to be successful we don't do maintenance a whining pipeline we just try and make it better it's not our job to make you logs to us so you have to ship your own you have to maintain that and we don't do I am off or anything like that we just do device so vendor Commandments we cannot these are the three things you have to meet all these and we won't bring the product in ship logs in a standard format because they spend a lot of time like kind of few I mean I worked around at seven but

if you watch they do say what they want everybody's logs to go into or all those things that unified you buy and that's fantastic but we don't want to live it so ship us your logs in a standard format we need to give us a quick and clean back out anybody else here has had this problem but I can think of two security products where six months down the road I'm still trying to kill this all in my environment and don't know how it got back in the first place and then expose an API to access your features who you see you you spent all this time making awesome things let us pull down the

information that we need so over the past 18 months we have spent a substantial amount of time improving blogging pipeline doing net flow summarization has been one of the biggest wins that we've seen to be reducing our AWS cost and we still get a full at least for TCP we get a full summary of what the connector that looks like you can't have easily do it UDP we're trying to solve it but at least we're not seeing a log of it generated multiple times both in the IDs and from the net flow themselves for each individual packet improving endpoint visibility this one is considered a win because we have basically nothing and a ton of

environments and when we kicked our lightweight stack out to just endpoints and developers didn't have a problem building code they wanted us to start moving it into their server environment where it can so we went from some visibility into corporate and almost nothing into production to you know I would say probably eighty to ninety percent disability where we wanted it so over 18 months that's that's really huge especially in developer trust which is the currency that is hard to come by standard echo space detection strategy spoiler it's attack matrix already attack everyone else do you think it's so high right now it's fantastic don't get too lost in metrics baseline include an indicator database so we use in this but if you

are doing anything with Fred and tell you can start taking in wolf or friend until feeds you're gonna want to indicate your database because you need to be able to go code around single pane of glass for analysts we finally got this certain vendors made it really hard but now one person can come in and everything comes in fully enriched and you can write well we only care about you making the first decision on what to do in this pane across you don't need to work investigation from here but everything that's presented to you you should be able to say all right the next step is this is this an incident or is this something I can close and if you

have to leave there leave this pane of glass to make your next decision we are failing ingest and rapidly act on threat Intel can't talk too much about this but we are proud of the fact that like whenever a couple of a doctor groups that they are tracking whenever their new stuff hits DTI or anywhere else within within 24 hours we have everything updated for us that we know if they're coming back there is CICP pipeline this has been a godsend specifically for carbon block rule II processor I find a you know how if I want to write a detection I make a pull request somebody approves it and within five minutes it's not only is it life

but he just kicked out across the network to the curb to see if the detection actually hits agree on standards for for production products you just standardized like you could write in whatever you want but don't you know don't expect your Perl script to make it in a product needs to be something that somebody else can maintain and then build relationships outside of security you know especially council that's a good one to have a friend in your legal team and then lessons learned okay email alerts suck please stop giving me email alerts so you wake up and it's like I have 9,000 where it's awesome not gonna check out nobody's going to check on find a

different way slack alerts is only slightly better but still called you I'd to control workflows and I hold that's why I didn't single pane of glass it is so painful to have to switch from your firewall to your next appliance anything else to try and get information don't do that just try and fork it into a one thing whether it ends up being like an orchestration solution or not figure out what works for you but don't live in multiple environments thinking capabilities instead of tools say I want full packet capture or not I walk Milwaukee it's ok to fail everybody exercise to blue team to fail so if you comfortable it if you're on the red team sucks to fail because ever

and then the blue team's like I can beat you it feels terrible so it's ok to fail and fail with your team unknown unknowns are still mysterious one of the biggest values that our red team has is letting us know where we are going like we're fine we feel much more comfortable if we know we are blind in an area then we have no idea why they're alright and another future so I can't put this back in the bottle AWS GCP Azure they're already here they're coming even more and so know how to work in these environments just you can't if developers say we love it you're getting it and then finally kinda ya figure bigger presence in China for

us specifically and getting forensic data across because every country has very weird security norms like Brazil and Korea if you want to access web banking cool you can basically gonna download a Trojan that guarantees that only the browser is running web browser processes access to anything this one time and if you have custom detection based off of what malware does congratulations you've just flagged all of Korea and Brazil as a malware that sucks so I hit time and that's actually the end of my presentation so we don't have time for questions cool if you have a question or two we have two minutes

[Applause]