BsidesLV 2024 - Ground Floor - Wednesday

[Music]

[Music]

[Music] a [Music] n

[Music] h

[Music]

[Music] [Music] [Music] n [Music]

[Music]

[Music] oh

[Music]

[Music] n [Music] [Music]

[Music]

[Music]

oh yeah [Music] [Music]

[Applause] [Music] he [Music] [Applause] he [Music] [Applause] [Music]

hey he

[Music]

[Music]

[Music]

[Music] TR

[Music] hey heyyy [Applause] [Music]

hey hey hey hey hey [Applause] [Music] [Music]

[Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] he [Music] [Music]

[Music] [Applause] [Music] he

[Music]

[Music]

oh

[Music]

w [Music]

[Music] oh

w [Music] [Applause] [Music] [Applause] [Music] I'm just Jing in I'm just jumping in [Music] something I'm just tring something I do you I'm just TR to give you something [Music] [Applause] [Music] [Music]

[Music] [Music] I'm just TR I'm just TR to give you something [Music] I'm just trying to give you something okay I do for you I'm just trying to give you something [Music] w

a

[Music]

[Music] [Music]

[Music]

is [Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

[Applause]

introduce you okay TR what was it be myant yeah me too okay right good morning everyone uh welcome to bides Las Vegas this is ground floor and I'm introducing our speaker for today um oh my gosh this is Blake Hudson do you mind if I tell him it's your first Y and this is Blake's first talk so I'm excited for him I hope you are too and the title of his talk will be pipeline pandemonium and a few announcements before we begin uh we'd like to thank our sponsors especially our Diamond sponsor Adobe and our gold sponsors Sim GP blue cat and Toyota it is their support along with our other sponsors donors and volunteers that make

this event possible these talks will be uh streamed live and as a courtesy to our speaker and all the other speakers and the rest of the audience we ask that you please check to make sure your cell phones are set to silence and without further Ado our [Music] speaker thanks okay welcome everybody to my talk pipeline pandemonium uh first things first uh you know these are all my opinions expressed by me not by my employer so who am I my name is Blake Hudson I am a offensive security engineer for PayPal I've been there roughly about two years now and I've kind of started off with the whole color wheel essentially of jobs originally was a blue teamer for a small

mssp in my local area quickly transitioned into red teaming for a Department of Education uh contractor and then from there have now transitioned over to uh PayPal as a full red teamer mostly doing a lot of purple team activities there I have roughly six plus years of experience within cyber security at this point uh really specializing in infrastructure and Cloud um security and technology and again this is my first time speaking at a conference so it's a little nerve-wracking to say the least um got this yeah thanks um I will say this is not something that I ever anticipated doing in my career and you know from a lot of friends family co-workers uh they all

kind of pushed me to do this and you know take the Deep dive share my knowledge with somebody and you know I honestly I recommend everybody else do this at some point in their career so all right let's jump right into it so what is a cicd pipeline well in real simple terms it is basically how uh to help developers uh make and push their code changes faster and with fewer errors out to their staging and production environment by automating a lot of that testing earlier in the software development life cycle and then the whole deployment process and the CI of that does stand for continuous integration and this involves basically having all of your developers kind of

use GitHub or some sort of git repository where they push all of their changes to in a shared repo and then they run automated tests on it to catch the errors earlier within the code and the software development life cycle the CD of that is I've seen it both ways continuous deployment or continuous delivery uh pretty interchangeable term but this is where once the software has actually passed these automated tests it is basically the automation of pushing your changes your code fixes to your staging or production environment and it really helps with that process because it helps deliver that code quickly and reliably to your end users kind of follows this standard you know pipeline here where you can see it

starts with planning what you're trying to do you code it build it test release deploy operate Monitor and it's this cyclical pattern where you know as you're monitoring it you you notice you have some sort of errors in there you go ahead and you know plan the next life cycle of it you code it you build it and it just repeats over and over so who are the kind of common providers that you're going to run into well Jenkins everybody knows Jenkins if you've done some sort of internal pentest before you've come across Jenkins you've probably exploited it dozens of times it is documented to death it is all over the place very very interesting attack paths there some of

the other ones that I run into a lot are Circle C uh Team City very minimally it's mainly kind of like a I I I feel like I see it used as a kind of like a test bed and then gitlab I see occasionally but the big one and the one that we're going to be focusing on today is GitHub actions i' in my experience I feel like a lot of organizations s are transitioning to GitHub actions because it's right there it's built into their GitHub repositories as it is there's no third party you know SAS application or other infrastructure that you have to stand up to interact with it it's just it's there it's enabled by default so

why not use it seems the easiest option so again some of the common uses for cicd pipelines in today's Major organizations are things that we've already kind of covered automated testing all of those code quality checks to catch the errors earlier uh Version Control your artifact management where are these things going to be pushed out to once they've been built and deployed and your entire workflow orchestration so the one that I skipped here though is IAC deployment and infrastructure as code um so typically a large organization uh who has a very large Cloud presence they're not going to go in and what's called click Ops they're going to go into the Management console click configure this click configure

that that's where a lot of kind of mistakes happen uh large organizations tend to have everything uh defined out in some software uh coded out very specifically for what they're trying to do to do everything at scale and at massive massive scale So today we're going to be talking about that infrastructure as code uh management portion and in particular again we're going to be talking about GitHub actions because I see that the most right now and particularly just AWS but it really can be applied to any of the cloud providers so obviously why do we want to test them it's pretty St right uh we want to identify any of the vulnerabilities misconfigurations within the organization's software deployment

life cycle and the cicd pipelines architecture components and processes it's also to really help prevent any sort of internal threat actors from poisoning any of the organization's dependencies or softwares or you know uh uh deploy deployments of code and uh during tons of different red teams internal uh Network engagements it's pretty common to Target software developers and get their level of access to where all of this is really possible so when do we actually test them me personally whenever I'm doing a a cloud assessment I always like to get my standard roles of view and read only for say AWS but because everything is typically defined in code somewhere say terraform State files things like that I

like to also ask for uh permission within their say git repositories or their cicd pipeline uh framework it just makes a lot of sense and in my experience with say cloud environments when you're doing a pen test you you will still find some misconfigurations here and there but I see the vast majority of the major exploit paths coming from the cicd pipeline and abusing that heavily uh there's obviously you could do CD pipeline specific assessments this is this is pretty rare though I don't get asked to do this very often uh but it is a way that you can do it red team engagements obviously this is a very huge Pathway to go and unfortunately in

my experience too there has been time and time again where organizations especially large ones aren't actually sending logs from GitHub to their you know their Sim or their CDC or sock I should say but there is so you're kind of operating in a blind environment where they're not really paying any sort of attention and it's just on the developers themselves to identify anything and then lastly uh internal Network assessments uh pretty common as well once I get sufficient privileges in inside of an environment I always like to try to Pivot to the cloud try to take that over as well kind of show the more risk with the fact that you're just trying to you know demonstrate more

value to that entire assessment who needs permissions right well turns out you should always ask for permissions before you start messing with these things because organizations can be very very sensitive with you interacting with these things uh kind of like I already discussed though uh when I'm doing a typical cloud assessment I like to look for or ask for that standard developer user within their repos and if they're using GitHub actions that's going to give you the access that you already need uh also asking for their standard pipeline software access so things like Jenkins if they're using a third party uh framework now a lot of organizations are going to kind of push back against that

they don't want you in there they don't want to get Grant you the access or they say you're a hacker you get in there um I always like to try to push back against that because try to really demonstrate and explain to them why why this is such a dangerous area and potentially very very vulnerable to privilege escalation or full Cloud compromise so a lot of these CCD Frameworks are going to be very very similar if you know how to exploit and manipulate one of these you can kind of take that same basic uh common knowledge or like ground level knowledge and transition it to one of the other providers pretty easily and quickly and

uh a lot of them operate off of vml files it's not going to be the exact same uh syntax but you can very quickly figure out what that is and then transition to that to to basically do the exact same thing that you're looking to do so let's actually look at some of the typical paths that I tend to exploit a lot of different ways so starting with the basics and the obvious uh once you get access to that git repository or all of their git repositories you want to do the simple surges and this is obviously a kind of a yeah absolutely uh this is not an extensive list by any means but the things that I like to look for the

of us access Keys GitHub personal access tokens huge this can be the end game for them just by gaining access to those uh GitHub personal access tokens Jenkins API credentials you could actually turn that API credentials into full console access to Jenkins with uh burp Suite Circle CI API credits uh as your client ID Secrets maybe you can authenticate as a um uh service principle gcp service account keys I see these up inside of get repositories just sitting there pushed up to a random repo and a lot of times they have things like editors permissions on the specific gcp project which grants you the ability to then start creating tokens for all other servic accounts uh obviously terraform Cloud

API key that can be extremely dangerous as well when you're searching for those uh I've been in plenty of engagements where I have just found that in a random variable inside of a git repository do a simple search and you find terraform Cloud API key you might have full admin of their terraform environment kind of game over at that point so the one that I skipped over here is runson this is a particular variable that I also like to look for as well this is in all of the Amo files for a GitHub action and this will indicate to you that they're actually using it so that search bar in GitHub itself is actually really really

good and I like to do all of these searches manually yes there's very good tools out there things like uh trule hog they do an amazing job of covering very large organizations quickly to find a lot of these secrets but I do feel like it misses things occasionally and it's just better in my opinion to go and do these searches manually so what's another way to kind of help you narrow it down uh well obviously you can look for these specific folders within their git organization to see what they're using things like Jenkins file or any of the dots circleci do GitHub harness and team City some of it might be Legacy and they're not using it anymore and I come

across a lot of repos where it will be like a circle CI and a GitHub folder and the same thing are they using both probably not but at least give you an idea of where you should be looking and what you should be looking for so some of the more interesting paths and this first one is I do it constantly uh I've probably done this thousands of times to steal secrets from every environment that I get into um so first off though with that open your eyes everything with a GitHub action is going to be almost right in front of you read through their yo files of their actual actions it will tell you so much information how are they

interacting with the cloud are they using standard roles open ID connect roles are they pulling in Secrets fromwhere somewhere uh are they pulling it from Secrets manager or from the actual repo itself are they assuming into other roles that might have different privileges all sorts of information that you can pick out specifically just from reading the EML files so the secrets where are they actually being stored well each cicd provider actually has specific locations where you store the secrets and in GitHub there's two locations either at the organization or the repo level and inside of the specific settings for that and then secrets and uh variables you can see that there's this highlighted uh box

there for Jenkins it's obviously the credentials uh URL for circleci it's going to be underneath the organization's context and then there's this environment variables gitlab it's going to be underneath the project settings uh cicd variables team City it's going to be underneath uh the actual projects parameters inside of environment variables and then harness.io another common one that's kind of up and cominging but it's going to be in two places either the project or the organization settings itself so we know where they're being stored now and this is where a lot of organizations are actually storing these things how do we actually get access to them well typically you can't even as an admin or a code owner of the repo you're if you

go to click the edit button for this you're just going to be overriding it so you can't see that clear text value of it however any of the pipelines that you actually configure to interact with it we'll be able to access those and that's the the pivot point where we're able to then see the clar text credentials so again reviewing all of the existing yl files that are in there you might be able to see that there is a specific format something that is like a variable name dollar sign curly bracket curly bracket secrets. variable name and that is going to be exactly what you want to be looking for so something to keep in mind too these secrets they

might be stored in there and no yaml files are actually calling them anymore and since you're not a code uh repo admin you're not going to be able to see that these secrets actually exist there it's also very very slow to do this manually going through every single repo looking to see if there's secrets in there being called by any of the yo files so I actually had chat GPT just spit out a script for me to see if it would work and I just had it spit out the Python 3 script that will take a GitHub personal access token because I come across those constantly and then iterate through every single repo it has

access to extracting out the secrets name for that repo and the variables names and when you run it it looks something like this you can see we've got two repos here where they are storing secrets and uh this is exactly the information that we need to start actually pulling these secrets out and using them so we're going to focus on this bottom one the C abuse as we could see that the secrets are typically also named exactly what you might think that they are AWS access key you know GitHub bot things like that so how do we now get to these well pretty simple you just go ahead and create a new Branch sorry uh very simply you just go ahead

and create a new branch of that repo that happens all the time developers are always creating new branches you add in your new malicious yaml uh file to that spefic specific Branch commit it to your branch profit it's pretty pretty simple um especially once you see what the ammo actually takes um you'll notice something here though I didn't say that you need so any sort of social engineering you don't need to convince someone to approve a pull request or even review any of your code changes you're just pushing to your own branch and executing things so this is what it looks like uh so we knew that cscd abuse um repo was vulnerable and we

went ahead created our new Branch created this extract. yaml file and then kind of stepping through this the name you can name it whatever you want sometimes you want to probably name it something that's going to blend in a little bit better compared to their other actions uh the second line there on push that's the trigger point this is saying as soon as I commit it to my Branch it's going to execute this action and do something perform something now once we get into the jobs here you can see this environment variables and this AWS access key AWS Secret key that's the exact format that you're looking for to identify that they're storing Secrets within their repository and they can be

just pulled straight into your environment variables of your Runner uh this runs on key that's what operating system that you're going to be executing all of this on and then we'll we'll kind of jump into the rest of this here in a second so you go ahead and Commit This to your repo uh your branch of the repo and if you just try to do something really simple like Echo out those secrets well each of the providers are going to be fairly decent at identifying that that that is actually a secret and stop you and they're going to mask it how how difficult is it to actually get around that and then just start seeing the clear text Keys it

turns out very very simple they have a very very rudimentary masking technique so first one here we just what if we print out the keys one by one all right yeah you can do that very slow uh what do we know about AWS access key IDs they're all alpha numeric and uppercase what if we just simply text manipulate it and lowercase everything doesn't catch that as a secret you now have the clear text access um we can print out the secret Keys uh split it in half cut print the first line and then on the second line print the second half and basically any text manipulation will get past their secrets identification uh my favorite the last

one I just like to B 64 encode it spit that out for you copy it out decode it it's yours something to keep in mind though um if they're using uh for instance you can see here that runs on auntu latest that is actually saying that it is using the default GitHub actions Runner um if there is something that's named differently than that kind of that format you know they're very likely using custom Runners something that they're they have up in say the cloud somewhere they're pulling it down it executes the action that you're trying to do and then it destroys itself if that's the case you really want to get in the habit of dumping out

all of the environment variables because you don't know what else has already hardcoded into these environment variables for that particular container I've found terraform Cloud API keys in there that got me full admin of their terraform environment I've found other GitHub personal access tokens that gave me admin access to several repos tons of potentially really good information in there so what about the situation where the organization doesn't want to give you access to their GitHub oh actually I skipped ahead there uh so this is just to show you if you Bas 64 decode that lob from the environment variables you can see we have all that sorts of information in there some of it might be

useful but you can see that we have the AWS access key and the secret key and clear text perfect now we can pull this into our own CLI start using that however we want um so in a situation where they don't want to give you access to their GitHub repositories it happens uh I have had to do this before in a real engagement I had basically chat GPT spit out another Python 3 script for me that will you point it at a a code owner or a organization a specific repository and then it's going to go and create the new Branch for you then it's going to create your malicious workflow file or action and then basically you commit it it push

uh gets pushed to your branch executes immediately and this one in particular obviously since you don't have access you won't be able to see this console output this is just to show you what it looks like and because you don't have console access you can't see the output of the actual action running so you wouldn't be able to steal the secret but in this case do the exact same thing pull in all of the secrets into the environment variables and then just base 64 and code the environment variables to a text file and then curl post request it to a system that you have on the cloud that's listening now you have all of the environment variables Bas 64

decode that and you have full access to all of those Secrets as well even though you never had console access to their GitHub be mindful of that though you are sending Secrets out across the internet uh that may not a good idea uh there's probably safer ways of doing that just kind of get your head moving in the right direction of how you can get these if you don't have access it's pretty interesting too because you know you just started off with a GitHub personal access token use the first script to iterate through see where the secrets are Target them specifically run the second one and steal everything so that's if there are storing secrets in the actual uh

repositories themselves a lot of organizations are going to use things like Secrets manager as well and and again reading through those yl files you might see that exactly and what Secrets they're pulling in and you could just copy that exact same code and put it into your own yaml file so here in my AWS lab I just created this prod secret key and I created again a new Branch with this yl workflow we're going to be pulling in those same Secrets because through our enumeration we found out that we have secrets manager access and then this highlighted box here this is all the steps it takes to then access their secrets manager assuming you have

those IIM per missions when you do this as well it's pulling in the secrets and then putting them right in the environment variables right for you to steal as well so just because they're being stored in a better location doesn't mean that you can't access them we're going to base 64 encode these steal all those and you can see that we got uh access to that prod secret key super secret key and super secret password so we can steal secrets from all over the place at this point uh Secrets manager there's ways to do it for hashy court vault as well uh gcp Secrets manager all really the exact same thing so what else can we do what's next

well again the common theme read through their other yaml files you might notice that this organization is trying to do things kind of correctly or at least more secure they have a a typical role that they're executing say low-level permissions with and then they are assuming into another role that's where they're doing more administrative tasks create new users uh assign you know security policies things like that so reading through their yam files you might notice that hey we really want to assume into this other role and through our numeration we figured out that we could assume into this particular role this deployer prod again this is all it takes to do that uh that uses line AWS

actions configure AWS credentials that is actually a GitHub repository you can go there read through all the documentation and see exactly how to use this as you might need to adjust things depending on your environment we go ahead and Commit This to our repo our Branch specifically and we can see that first we are running as that deployer prod role or pre-employ we have successfully assumed in and then we became that uh full deployer prod and through our enumeration we figured out that this is a full admin and this is where you can kind of go two ways you can continue to just issue commands directly here in the pipeline and to achieve whatever you're trying your goal

is or pull out these secrets in the environment variables and put that into your own awsi and have access for that for about an hour typically roles are configured at a minimum for an hour and uh at most for 12 hours but I've never seen more than an hour so uh let's just continue issuing AWS commands directly here inside of our pipeline issuing three more commands you could just go ahead and create yourself a new user attach that IIM admin policy to it and then set a password now we should be able to have console access as a full admin just to confirm yep it got pushed up to our specific Branch we have that

new offs SEC user with admin access and we could log into that console a lot of times it's just a lot easier navigating through the console as uh you know to view certain things it's just a lot easier than the CLI so some of the actual restrictions that you might run into and this is one that I've actually come across quite a few times um when you create a role you have this tab for trust relationships and in this case I created a open ID connect Ro so it's basically allowing uh anything from GitHub to assume Into Your Role uh if it meets certain conditions and in the output here you can see that one of the conditions the top one is

repo and that is bit Legion that's going to be the organization cicd abuse the repo itself and then poll request and so you know to interact with this role it has to be coming from a poll request and then the second line is defining that it has to go into the main branch pretty standard and honestly the first time I came across this I thought I was dead in the water I thought there's well I can't get this key uh I'm not going to be able to social engineer Somebody To You Know approve a poll request but this doesn't actually say anything about actually getting it approved you just need to submit a poll request so if we try to actually execute

this um it will fail once we commit this up to our Branch we can see that the actual action failed well that's mainly because we didn't meet any of those conditions the action executed before a poll request was opened up and you know so that condition wasn't met so how do we actually get around that well it turns out GitHub has a lot of different triggers and reading through these you might have one that will specifically um show you your use case and when you're uh for this particular one there is a trigger for on poll request so nothing will execute until you actually open up your poll request that sounds like it's going to be perfect for our specific uh

scenario go ahead and Commit This to our branch and and uh open up our poll requests once that's open it automatically executes our GitHub action and you can see hey we met all the conditions we now have access to this role again you can start issuing commands through the cicd pipeline by adding to your yaml or just uh pull out these secrets again and Bas 64 decode it put it into your own CLI use them all you want there for the next hour now on say like a red team submitting multiple PLL requests that might be pretty loud uh typically a lot of places are going to get send out emails like hey there's uh code review

that needs to happen and you're likely going to get caught even if you go in and close it out real quick you might skir it under the radar but it might be easier or better to just go ahead and pull these keys out and use them externally at that point um and this is just to prove you know uh I had a thought at least the first time when I pulled these secrets out well it has this specific trust relationship so you can't use these keys out outside of the actual pipeline that's not true you already met the conditions and those keys were granted and they're valid for an hour so you can pull them out and put them into your own

CLI and continue to use them so this next one is one that I actually came across uh in my Consulting days and it's it's pretty interesting you're you'll probably notice almost immediately what's wrong with it and why it's really bad they were trying to do the right thing and make their trust relationship really more strict to interact with their roles but they ended up doing pretty much the the exact opposite so this open ID connect role you can see the trust relationship itself is uh repo bit Legion star/ star colon star that is basically saying the organization star so the organization has a prefix what repo any repo because of the wild card and then what branch

any branch sure some of you already know exactly what's wrong with that but let's kind of play with that so in this particular engagement I just went ahead and thought all right well any branch within that organization or any repo so I created an arbitrary repo created my own um yl file and uh committed it to the branch which then assumed into that open ID connect role successful yeah we met the condition pretty simple right but the real interesting thing from there is that wild card for the organization since it was bit Legion star that's that's a prefix I can put anything you know I created a whole new basic um Gmail account with bit Legion

test that fulfills that condition and then any repo from there and an any branch so created my external repo with a again an organization that's not uh familiar or has any sort of relationship with the main organization go ahead create this yaml file commit it to my my specific repo and we can see that we had full access to that as well so this is pretty bad because they're allowing a unknown completely unrelated GitHub um repository interacting with their AWS account and this is a great form of persistence and um you know some random repo out on the internet can be interacting with your AWS environment and all based off of one wild card misconfiguration so you might be

thinking how do we actually go through each and every one of these roles to see what the trust relationships is that's very difficult timec consuming to do manually if you guys have never used it uh big shout out to Bishop Fox for their tool Cloud Fox it is awesome at doing all sorts of uh situational awareness for cloud environments and this one spits out a ton of information specifically one file that shows you these trusted subjects where you can see the the actual role name it's provider so it's open ID connect provider and then these uh trusted subjects that's exactly the conditions that we are looking at uh for each of those and then is the role in admin or not so you this

really will narrow down and help you target specific roles within the environment so again huge shout out to Bishop Fox uh and then just for fun this is one that I have done in the past this was a particular engagement where I got uh access to their git repositories and they had one in in particular for their entire terraform uh scripts and I gained access to roughly 15 or 16 different GitHub personal access tokens and was trying to explain to them that I had full control over their terraform uh repo they didn't believe me and they wanted me to actually try to do something try to push a change so what you can do is once you get your hands on

a lot of these GitHub personal access tokens well you might be able to they might have sufficient privileges to actually do something in this case the organization required three different code approvers and then the own owner to then merge it into the the main branch turns out I actually had enough keys to do that so in my own Branch I went ahead and just updated the uh um readme file up I put updating this readme at the very bottom of the uh the file committed it up and opened up a poll request then I went back and created a new action to execute these curl commands and as soon as I push this up to my repo it's going

to each of the GitHub access tokens needed was going to approve and then I'm going to hit all of my approvals and then the last one is going to be to then uh merge it into main yes these are just Cur requests you can do this in the CLI but I thought it was much more interesting to kind of demo it in there actual cicd pipeline what it looks like in the console output is you can see that it was approved and then successfully merged and I was able to actually update their readme file uh so I I was at that point able to verify and prove out to them that I could change basically

anything within their terraform repository fully compromising their Cloud um have done it before it's it's been a while since I've was able to do that but it's a really interesting attack path so how do we actually protect ourselves well I I feel like a lot more research needs to be done into this and I I think the cicd pipeline providers need to be a little bit better themselves about identifying Secrets but it does come down to being very very mindful about what I am permissions you're assigning to each of your roles and it really comes down to strictly adhering to that principle of lease privilege you don't you really want the roles to be sufficiently you know

provision to exactly the roles or the task that they're set to perform uh if they're just moving files around from S3 bucket to bucket there's no reason to give them IIM permissions whatsoever one of the things you want to consider too is uh do do your developers really need access to every single repository and that's usually a blanket thing if you're a developer you're going to have access to every repo in the organization really unnecessary um and because with that you might be able to use specific Runners from different repos into different GitHub repos as well so there's you got to have some sort of delineation to show that this specific Runner is for say the

finance team and this Runner is specifically for an infrastructure team or web app team you don't want to cross use runners or GitHub repositories for several different tasks these uh creating roles with restrictive trust relationships uh I haven't come across one that was really really great that fully blocked me but uh I think a lot more research needs to be done into that of really how to lock it down to a specific action and really you don't want it it's it's kind of a hard line too because you don't want to restrict developers from doing their jobs by putting too many Hoops in front of them but you also have to balance that with the security of it and make

the um the actual action itself have a specific trust relationship so it blocks anyone from coming in and just executing any of these and stealing the keys something else you can do is enable AWS guard Duty and this will use Ai and machine learning to help identify if the credentials are being used in any sort of suspicious way in my experience though this is pretty hit or it's missed a lot of things that I've done uh with some of these access Keys it's supposed to be able to see that hey it's not coming from a GitHub action anymore it's being used in a CLI somewhere and it's no longer usable um so it's it's been

real Hit or Miss then you can also um limit the lifetimes rolls session tokens obviously the default is always going to be an hour or 60 Minutes at the at the minimum but never set it for 12 hours that's pretty unnecessary and that does bring us to the end so I kind of flew through that but does anybody have any questions

yeah thanks great talk um the a lot of this seems to be leveraged on getting access to exfiltrating the GitHub secrets and using uh thirdparty Branch outside of the main repo to get access to those I don't have a ton of experience but I thought that those secrets were protected from external repos so maybe you could talk about how that works and how you might defend against that if they're not indeed protected from third party repos so say that like that last trust relationship that I showed that had the the Wild Card attached to it is that what you well that was having to do with uh getting access to the IM am stuff but this is more earlier in the

talk where you talked about exfiltrating GitHub secrets in Bas 64 to bypass the exfiltration protections MH but I was like I didn't think you had access to those Secrets as a third party repo when you're running the CI well no so you're actually running it inside the organization you're just creating a new branch of one of their repos that is storing the secrets itself so then is this more of an Insider threat type okay okay great I missed that yeah so then with an Insider threat scenario is there a way of Defending against that unfortunately I like I kind of said there needs to be a lot more research into this kind of protections um

personally I think the providers themselves or the Frameworks need to be much better about identifying the secrets I've talked to GitHub about it before and they their response was essentially well we can't prevent people from willingly exposing secrets so they don't necessarily Environ so you can environment protections which is similar to so you can use environment protections as uh well the secret can only be available for example for the environment production M and then you have to um run it on the main branch or in the production environment and in that way you can uh protect at least against just random branches accessing the secrets okay yeah I come across that in my experience it's seems like a lot of

organizations just kind of leave everything open and the thought process is oh you have to be a developer or have very high privileged access just to get into the git repositories or be able to execute things so they just they lean on that a little bit too much but good to know thank

you so sorry if I'm barding the mic um but in that scenario how do you do CI because the point of CI is to prevent errors before entering in the main branch so if it only runs in main then it's too late and your bugs or errors or you know attacks from the red team have entered into main it depends yeah yeah that's a good

point any other questions don't be shy

you mentioned getting caught through uh uh poll requests that triggered like an email to go review how else have you been caught doing these you know I I can't even say that that poll request one actually did get me caught until probably months down the line when someone reached out to me for the most part I kind of talked about it a little bit um a lot of organizations in my experience aren't pushing their logs from GitHub to their sock their seam and it's kind of just on the developers laps to try to catch all of that which they're not doing so a lot of times you can operate kind of uncontested in that environment and not A lot's going to

catch you um I've done it time and time again over the past couple of years and I've I don't think I've ever had anybody reach out to me except for months after the fact like hey was this you trying to do this oh yeah that's a that's a branch I left up there titled offs SEC on accident whoops but if you clean up after yourself too close out all your ped requests you might you might be able to just skirt completely under the radar for quite a

while any other questions I'll be I'll be sticking around too a little bit for if anybody has any questions after the fact now I saw people taking uh pictures of them slides so I know you guys were curious if I if I'm up here with social anxiety I know you guys can ask questions all right like you said he'll be up here to uh answer any questions yall have or chat so awesome thank you everybody thank [Applause] you oh

[Music]

[Music]

[Music] I [Music] oh [Music] n [Music]

[Music] [Music]

[Music] [Applause] [Music]

[Music] a oh [Music]

[Music]

[Music] a [Music] [Music] [Music] [Applause] [Music]

[Music]

[Music]

[Music] oh [Music]

[Applause] [Music] hey hey hey hey he hey [Music] [Applause] [Music]

he [Music]

he

[Music]

[Music]

[Music]

[Music] track [Music] St

[Music] he hey hey hey [Applause] [Music]

hey hey hey hey hey [Applause] [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music] he [Music]

[Music]

[Music]

he

[Music] h oh [Music] a [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just TR to give you something okay I do I'm just tring to give you [Music] something I'm just TR to something I do I'm just TR to something [Music] he [Music] w

[Music]

[Music] [Music] I'm just I'm just TR to give [Music] something I'm just to give [Music]

something good morning everyone everyone ready welcome to besides Las Vegas ground floor and this talk is building data driven access with the tools you have and this will be given by John Evans this is ground floor so you know people uh first talk so we all are going to be very supportive but before John's able to start I going to have a few announcements we'd like to thank our sponsors especially our Diamond sponsor Adobe and our gold sponsors prism Cloud blue cat and conductor 1 it's their support along with our other sponsors donors and volunteers that make this event possible these talks are being streamed live and as a courtesy to our speakers and audience we ask that you please make

sure to check that your cell phones are set to silence otherwise the will will be consequences I just made that up I'm sorry no consequences um with all of that let's get started and please welcome John Evans Cool okay um so I'm I'm actually G to just put the microphone here everyone can hear me at the back sounds great um yeah data driven access um so before I get into the uh the meat of this presentation and what we're going to be talking about just a little intro of me um so yeah who am I um that's what I generally like to do during the summer on weekends when I inevitably on a Monday call my boss and say I've damaged

myself in some particular way um which is downhill mountain biking so um but I haven't damaged my hands yet so I'm still good to work um so who am I I'm uh 10 years experience in corporate it um I actually joined the IT world from the Apple Store where my job title was lead genius um I've been um so right now I um work for company called Ceda uh Ceda I am a engineering manager of technical operations um few things report up into me uh it and devops um we call it Cloud engineering but I thought i' just you know um so yes so C's a healthcare technology company uh modernizing healthcare provider billing systems um don't let the accident fall

you I am from the UK but I live in Denver and despite working for a Healthcare company I still do not understand the US system um which is why I'm glad we do what we do so um one of my credentials for talking about this today uh because as you can see I am the evil manager word um well I've lived the journey across wide range of IM Technologies uh from a company that had a custom database abstraction with multiple feeds to active directory L app a custom shof implementation that you had to write XML to get new um uh providers added um to where we are V Ceda today which is kind of a we've got all the toys uh an Octan

implementation with all the best case scenarios so this journey from you know what it used to be to what we have now is um made me really passionate about access IM am and and uh SSO and stuff like that so um so yeah um well else I've got ADHD which means I've been freaking out about this talk for two months and finished it about 15 minutes ago um and as this is my uh first ever conference talk other than you two only two others have seen this my wife and my daugh uh my dog Bobby um she didn't have much to say about this um but I thought if this sucks at least I put a picture of

my dog up um so you can't be mad at me um my wife's reaction um you know she usually reacts when I tell her an intricate detail about something she doesn't care about sounds great um but then she came back and was like wait a second this seems obvious why doesn't why does it take my company so long to terminate user accounts and remove access and stuff like that so um onto the talk I guess um so yeah what is it what what do I see data driven access as um so in short data driven access gives you time gives you resources and removes the burden of access from it over to the people equipped to make

those decisions um we use as much data as we can to drive automated access decisions and then we fill in the gaps um using delegated access management to empower system owners to Grant access to the tools they own um so what what are we going to talk about today um I'm going to basically go through a journey of our IM am stack and the lessons that we learn from implementing it um if some of this seems obvious then that's great um but if you come back if you come away with even one thing from this talk then I think we're good I think we're successful um I was actually at Octane a couple years back and uh was watching a

similar talk from from Netflix and I was like this seems obvious and then um they mentioned one thing that actually uh completely filled a huge hole in my offboarding strategy so that was definitely worth it um I don't know if some of you were looking at this and you know we're going through some of the tools that we use and being you're thinking this is impossible um you know hoping to show that there are ways to you know manipulate the tools into doing what you need them to do um I don't want this to be a Sal pitch for any specific tool um there are loads of tools out there I can't talk about specific ones but I'm hoping to

Pro provide some inspiration for what you're using um hoping this isn't a brag um I've been where you are so um yeah let's look at the Ceda like a lightweight view of Ceda stack um so I've put perfect in quotations um because I think that we all know that you know U we're always going to see what needs to be fixed and What needs to and what is broken but I was at the sphere last night and saw a pixel out because that's what I'm like so I just put that picture there because it kind of reminded me of what you know perfect actually is um but I'm going to go through the stack talk

about the tools um try and give you some some ideas that can relate to your environments um at Ceda we tell prospective candidates we you know one good thing about working here is we have all the toys um provided you make a good use case for it use them to their fullest we're we're we're happy to you know use a tool an effective tool for for the use case cool okay um so I'll start with our source of Truth um and I've heard this sometimes that active directories are source of truth I is our source of Truth um I am of the belief that something like a HR System is a source of Truth um and really um HR systems are

you know kind of a big deal um they're the systems that get people paid so there's a lot of control there there's a lot of auditability there's a lot of um uh you know actual control in how you grant that access so I always want to start with um a HR System as a source of Truth um and when we try and get the data from a HR System I'm I'm kind of saying that we're looking to get as much data resolution as possible um so some examples of like you know why I class to be good layers of data resolution you know we' got the easy stuff name Department title who your manager is um you know medium stuff

location office um people are remote nowadays knowing if their remote is actually really important um to some of the more kind of esoteric examples that really do help you with automation um are they a people manager you know job title is not enough to judge if someone as a people manager um sometimes we have to say does this person have more than one person report or more than zero people reporting into them if so they're a people manager they could have manager in their job title but it doesn't necessarily mean that they're a people manager um and we use um Rippling at Ceda and I think that the the manager question was actually really key for us getting

involved with the people team to um pick a decent system that we could you know automate off of because we kind of you know I sold it as a well we can add managers to all of the email groups to all of the slack groups to all of the things that they need to use from day you know from from Day Zero like wouldn't that be cool you don't have to open tickets um so I think like you know our goal is for someone to have access to all the core systems they need from day Zero um so for me um you know it was really important to uh relate that to the hrstem HR to the HR team the people

team um so that was a very very like quick kind of look at the source of of of Truth um but I'm going to go back to that in a second um I also want to talk about source of data uh because um I think that you may also hear that o like we use OCTA I think you might also hear that that is a source of Truth um but I don't see it that way um and for um a few different reasons um I think that you know modern IDP should have a lot of these features um but a rich tool set of rules and automations and things that come out the box that will let you

actually um Drive some of the key automations off that source of Truth data um is really important in a source of data and I also think that you know source of Truth is designed to be mutated in the actual you know SCE itself um a source of data we can build Integrations outside of it um so I'm going to go into some examples of kind of what we've done to leverage octar as a source of data and and what you know you could potentially do as well if you have a source of data that has a API um so yeah let's look at some examples of that I've got a pretty good um example here um

so this is an example example of changing our source of data outside of OCTA if OCTA was our source of Truth we wouldn't be able to change it outside of you know what is already existing um so this is an example of uh if someone's on call um we run a Lambda um on a schedule that pings the page of Duty API is this person on call um yes so you know add them to an ox group hit the octra API add them to an Octor group and put them into the on call group that then fires a bunch of rules that essentially say okay this person's on call they get added to a slack Group which gives them the

handle of on call they get added to a oct to a um AWS Group which elevates their access while they're on call um which you know I think is a is a a good way of automating um and keeping our access as low as possible but having an automation that elevates it if needed um so these Downstream systems um AWS Slack are we using a bunch of like API Integrations with an OCTA not really um we're just doing a lot of rule logic in Octa um and I think this is one of the things that I'm I'm really passionate about um there are vendors out there that promise hundreds of custom Integrations um and they're uh you know

they they they do have a high value um I always think that you should try and do what you can in your platform um because if you have to change vendor if you have to change to a different system the rules will still exist and can be easily changed and modified you know there are vendors out there that that will do hundreds and thousands of different Integrations like when we looked at that vendor it was about the same cost as an engineer um and I don't think that we have an engineer working 308 hours a week on this um but your situation it may work but for us um the rule logic works really well um now it does rely on the

downstream application being a very good SSO citizen um and I'll talk more about that later um but if you have that and the rules provide all the access you need then you can go even further we do kind of more advanced rle logic um this is something that I've called exclusion groups exclusion groups um let you use the octar rule logic to say if the person is a member of this group and also not a member of an exclusion group then you can grant access if they are an exclusion group don't Grant access and we have situations where a user might be an exclusion group because they're in a different country country um they can't

access the data outside of the us or if something happens and we have to limit access we can build models out of those exclusion groups to reduce access without removing the person from from OCTA um so you can take rule logic and this is an OCTA but you know look at your IDP look at your system to see how far you can take rule logic because um you can go pretty far with it and obviously sort of Truth all of these items can be used in rule logic and that's how we get a Boolean that gives you know makes someone a manager and they get added to those groups and slack groups and things um be careful a bit of caution um

I was cleaning up my own mess last week because um I was using a manager field manager Fields can change a lot easier than like a depart we have like a really good process for departments if a manager leaves the company then you have to clean up that mess which is what I was doing last week um because it's a lot quicker for someone to leave the company than it is to change the department so a word of warning if you are using you know fields to automate be careful of using manager job title things that can change easy um because yeah you'll be cleaning up a mess like I did um employee LIF cycle uh this is one

that you know I think that we're particularly proud of um and again this is using um Octor built-in or additional um workflows um when I joined Ceda um I'm boarding used to be you know the responsibility of the IT team a lot of manual work a lot of tickets granting access doing a lot of things uh to get people into systems um offboarding used to be a 40 step manual process um every time someone left the company that was a process that was prone to errors um you know it took a lot of time um we as a company need to be hiper compliant so we had to solve this and um this is a very dense diagram um but I

really want to kind of you know show what you can do and this is in octus workflow module um when it comes to offboarding someone um or onboarding somebody um just some very quick you know looks at the onboarding process um we are able to uh take consume a onboarding request from the hris so nobody is creating the account manually the HRS does that when somebody starts uh in Octa work flows we start to um create uh we use some of the built-in modules to create a system of record um to start logging what applications that person has access to and then we even have custom API steps as well of the arboring um so I think like one cool

thing that one of my team implemented a couple weeks back was a API call to purchase a Google license if we don't have any Google licenses buy the license for that person that's it's all automated and then evidence generation at the end because we are kind of responsible for being audited um the offboarding process 40 different offboarding steps have been kind of condensed into this um uh flow which again is um it is not responsible for offboarding somebody um we're not getting pinged and being like hey this person's leaving can you make sure to deactivate their account on this date at this time we get an API call from our HR System and at the appropriate time they're off

boarded um and then this executes um I think some really cool things from this um which is uh you know off board is never nice but there are some interesting things here um so after the OCTA workflow step um we reach out to different systems depending on what makes sense so um octa's log for offboarding Google licenses isn't as good as another tool that we use called lumos which I'll go into in a second um uh so we make an API call to lumos to do a more robust Google offboarding process you have to keep a license active for 30 days and then we have to remind people after that time I don't think we're quite perfect there but

that's the that's the vision um and then we uh hit a system called retriever if that person is remote um then we will hit a service called retriever which sends a box and un label to that person to send their um um item back and that saved hours of uh my team being in um uh my team being at FedEx sending boxes to people for them to send their machines back so um really really useful there um I'm sure if you can think of any other systems that you can make a custom API call to then you can extend like what it is your system does um final step of the process um delegated access management so this

one of the things that I'm also really passionate about um you know who is providing access to your systems so we've done the data we've got as much Birthright access as possible um we've done the automations where possible now you know not every tool is um something that someone should have access from Day Zero so who is granting that access is it it are they equipped to do that like do they understand the level of access that they're granting do they have to run around and and get approvals and you know get that those kind of things or can we move that over to a uh delegated access management system um like a system like figma like you know our

design team knows who needs access to figma I don't I have no idea designers I guess um but they as owners of the system will know who needs access uh people can request access that can get approved by a manager and then that access is granted upon acceptance so no tickets being created um we use a tool called lumos um which is a pretty good Tool uh to do this um and I think that once we introduce Lumas for access requests we dropped our it tickets by about 2,000 over the course of a year um which was huge for the team um so yeah um so that's like a really quick rundown of um our stack and you

know where we've come to and some of the challenges we've gone through um but yeah as I said I know change is hard um we had to to you know get to this point it was a journey um and this wasn't intended to be a brag about you know oh how great we are or whatever um when I joined Ceda we had a uh kind of semi-automated system where um the HR System didn't integrate with anything um and the HR the people team had to drop a Google like a CSV onto a Google Drive which would then be consumed by a Lambda and then fed into the Octor API um so um we didn't have delegated access

management 40 step manual offboarding process so all of these things we've had to um you know come AC uh get over to get to where we are today um and um yeah so I'm going to finish this out with some of the challenges that that I am very aware of um SSO probably hear from vendors yeah we do we do SSO it was actually mean to do SSO now this is a polite slide in my internal meetings it's a lot more swear and angry um but you know this is some of the problems you can have with what constitutes SSO like you have to have the Enterprise Plus+ super mega OMFG plan um or you

have to um uh you know SSO tax um we do or loging we don't do provisioning so you know is your team buying a system where somebody's going to have to create individual users um you know we we do provisioning we don't do group sync so they're going to have to move them into groups manually um I've actually been on calls with vendors telling them how to implement SSO um and it's surprising how many don't um so um I understand that SSO can be a big problem especially if you're getting requests from teams to I want this tool I need this tool um so some tips really uh you know be involved with procurement um it's not

the most interesting thing but you have to be in the step um to help the teams understand you know why what they might want is going to cause problems um you know if they have a system that needs 400 people in it explain why they're going to have to have the Enterprise plan um or they're going to have to try another vendor um but also work with them and you know how many is it a critical system how critical is the data how many users does it have does it have five people and it's like not that critical then maybe the team can manage it themselves and I think that you know what we've seen from

a lot of the automations that we've done previously is that if we have to build a custom feed then maybe we can is the implementation cost going to be worth it not sure but it's very you know building a Lambda that consumes an API that drops an SFTP file onto a onto a drive you know is is is possible um so you can you know get around it don't accept that you're going to have to go in and manually provision each user um I would you know hate for for us to have to do that um the world of HR systems I tried to find a you know the most modern HR System I could that is about right um

but yeah as you said previously um I know the pain of struggling with bad HRS systems um and that was really what got me involved with our people team to make the choice of Rippling and to understand you know if we had more capabilities we could help them even more um and before that we we had to make do and do this Lambda that you know consumed a CSV file dropped it onto Google drive it wasn't perfect but it did also kind of put the pain over to our people team which meant they were also invested in helping us uh get that done as well um and I think like as well changes you know it can also come from within um

really proud of my team that are not accepting how things are you want your teams to be lazy you want them to ask why am I doing this over and over again this makes no sense you know Empower your team to to not be helpless um and then give them you know support to go beyond whatever can be done in the UI can they code something custom you know um I'm really proud that we've got a fantastic team that are growing into Engineers by asking these questions um and a lot of what you saw was actually executed by the team I'm just here taking the credit for it because I'm the MW um so we wouldn't be where we were

without the team um and get the data you know if you're spending let's say if you're spending X time at FedEx shipping boxes then if you're getting a system that does that automatically then you're going to save why amount of uh of of time um and yeah you know as I said from the beginning um I hope that you've taken something away from this today if you have I'll consider it a success um datadriven automation gives you time resources and removes the burden of access from it um over to people that can make those decisions um look at the tools you have uh do you need the all singing or dancing 100 integration application management platform or can you do

something with what you've got with a simple API call um and uh how can you make more from the data that you already have if you do have a good linkage with a system can you use that data to make and to do more automation um so yeah thanks for letting me talk about this and uh showing up I really appreciate it [Applause] ready for some questions then yeah anyone have any questions all right thank you hi there um how do you handle a source of truth that is a little too malleable like people have too much access to change things or um doesn't tie directly into like spending or earning because I know you mentioned

that at the start yeah I think this I actually heard someone ask this question last night as well um when I was kind of pitching this you like you're saying that your HR System is too malleable um and you know I think like that's tricky right and do I have a good answer for you other than to get involved with the team and explain why this is a problem and um you know if if that is something that you are using as a source of truth then it needs to be truthful right and not easily changed cuz um get involved with the team um if they are doing things like departmental changes get involved with putting a

process behind that so they inform you if they change that um because yeah you don't want them oh we'll just change the department and then everyone loses access um and I think we've done pretty well there but there's definitely work to do um yeah I wouldn't consider a sort of true to be that malleable but apparently it's the case

so this good stuff great presentation you showed um towards the beginning of your presentation a flowchart where you had a mapping defined between a person that is on this team at this time right their on call needs to inherit this access by being in these groups right so I we kind of think of that as a mapping between who you are and what job you have and what permissions you inherit now I think the one that you showed was relatively simple right it had about three outputs but in a more complex Enterprise you might have 10 15 20 outputs right so how do you take care of managing those mapping if you know your OCTA crashed and lost

everything right or you got hit by a bus right like how do you reconstruct all of that Stu for somebody that's not involved in this process one one thing I didn't show during this presentation um was uh I've kind of and and I didn't show it because we haven't fully implemented it yet but the I've got a document about basically making Nam spaces so um there's a uh philosophy well yeah at least a philosophy of this is the type you know app colon application Name colon access level that's the kind of group that has access to the application and then you have an account collection that will drive that and then you have like exclusion groups ex colon Etc so um I

didn't put that into this presentation because uh it prob it be a bit too dense but um I have got a doc that I'm working on to try and show that in more uh and but I want my team to also implement it before I start to shout about how cool it is uh but yeah like a n basically I'm making like n spaces with Octor

groups hey uh nice presentation um I think that you need to have a certain amount of maturity in the company for doing things like this what do you think about all the groups that are involved in this for because maybe it is has more maturity than HR or processes internally what do you think yeah I think like as well um I'd say it's like obviously our implementation isn't perfect there's definitely stuff that's in it like that's in Octa that that needs to be cleaned up Tech debt um I think that we've kind of matured at the same rate as our people team and um if that isn't the case then you know I think that um

work to I know I keep saying it I know it sounds you know um rich but work with them to understand that like if you can get this right then all of this stuff that frustrates you we can automate it and I keep going back to like email groups like we're not there we're not perfect on the email groups yet but they understand that like if we can get more data I can they're not going to have to add people into a Google group they're not going to have to automate that um offboarding they're not going to have to create a ticket for it to offboard people right they can handle that themselves they're not going to have to try and find an IT

person to do it um yeah I think like and maybe that maybe we're lucky with that that our people team have matured at kind of the same rate um but yeah as I said like try and sell the what can happen if you get it right any other questions all

right so uh in my organization we're uh starting an exercise about relabeling our D data with you know the evolution of of pii laws um do you have any sort of like in systems or mappings in in your um in this infrastructure to to sort of say okay this is something that needs I think you mentioned needs to stay in the US or otherwise and and how did you go about that um we go off standard Hippa um the systems it's basically I think standardized with HIPPA which is you know Green systems are do not contain P you know Phi um yellow systems could be um and I'm looking at Aaron red systems do have um Phi in them and we do have to

control access yeah this was uh one of the early things that we did from the security side and not the it side was try and simplify access down to red yellow green um uh maybe haven't explained it perfectly uh but uh uh red is uh sensitive patient data yellow is sensitive data that is not patient data so this will be employee data financial data things like that and green is lunch menus I don't care if it's on the internet um your team you know obviously each company's different in terms of the access levels that they need to do but we focused on just that really simple like red yellow green Traffic Light thing to start um

and there are in fact rules in Octa named you know red us only red and so on and we can use the Octor rules to map team constraints and application constraints to those um those labels thank you so the the kiss keep it simple silly any other questions oh yes

so I mean great perspective on datadriven access right and and one of the things that um we always speak to when we uh talk about access is entitlements U and the way entitlements feed into your entitlements repository and the way they Branch out into the systems there is always a delay component involved right I mean so you know maybe your entitlements refresh happens overnight flows to down Downstream systems maybe the next day whatever right so how do you uh bridge the gap between the time you know an access is supposed to be revoked um you know till it gets to your entitlement repositories and those individual entitlements are truly reflected down right cuz to avoid failure such as bad

connections from somebody or you know too many unauthentic unsuccessful attempts cuz Hey that service ID is still tied to that person and in um you know your system of record and that causes trouble down the line so how what's your u i mean is there I mean given the cloud approach I think it's more Cloud Centric but is there a Magic Bullet to to I think that goes back to the uh SSO being a good SSO citizen um and I don't think everything's perfect there so if all else fails we obviously do remove login by losing SSO and you can't log in without that um but if an application is a good SSO citizen Octor is really good

at essentially when that person's access is revoked it will fire Sim calls for every person that every application that person has access to which if your application is a good s associat is then it will remove the access um and I think that we we also the security team have done a really good job um with lumos as a delegated access management platform which has a also a temporary amount of access time so we do just in time access for AWS access um elevation and that also does the same thing where because it's a good citizen it removes the access um when that time is is is up but it's not perfect there are systems that

people still existing but uh fortunately for the most important ones it does actually remove access any other questions I'm glad you left a decent amount of time for questions everyone wanted to know more so I just want to thank you for your talk and um big round of applause everyone

[Music]

[Music] o [Music]

[Music]

[Music]

[Music]

he

[Music] h

[Music]

[Music] [Applause] w [Music] [Applause] [Music] I'm just do I'm just TR to give you [Music] something I'm just TR to something I'm just tring to give something [Music] m [Music] [Applause]

[Music]

[Music] [Music] I'm just try to give you something I do you I'm just trying to give you [Music] something I'm [Music] just I'm just try to give you something [Music] oh [Music] w

[Music]

[Music]

[Music] [Music]

[Music] is

t

[Music]

[Music] [Applause]

[Music]

[Music] [Music] oh a [Applause]

[Music]

[Music]

[Music]

[Music]

[Music] a

[Music] l [Music]

[Music]

[Music] he [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music] n [Music]

[Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

[Music]

[Applause] [Music] hey hey hey he he [Music] [Applause] [Music] [Applause] [Music] he [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey he [Applause] [Music]

hey hey hey hey hey hey [Applause] [Music]

[Music]

[Music] [Applause] [Music]

[Music] for

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

he

[Music]

[Music] you [Music]

[Music] [Applause] w [Music] [Applause] [Music] I'm just TR to give something I I'm just TR to give [Music] something I'm just TR to I do I'm just trying to give you something [Music] w

[Music]

[Music] [Music] I'm just TR to I'm just trying to give you [Music] something I'm just to give you something do I'm just trying to give you something [Music] w

[Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] be [Applause]

oh [Music]

[Music] [Music]

[Applause]

[Music]

he

[Music]

[Music]

[Music] e [Music] h [Music] he

[Music]

n

[Music] [Music]

[Music] [Applause] [Music]

[Music]

a [Music]

[Music]

[Music] n [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

[Music]

[Applause] [Music] he he he he [Music] [Applause] [Music] [Applause] [Music] heah [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey [Applause] [Music] hey hey hey hey hey [Applause] [Music] he [Music]

[Music]

a [Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music] he

[Music]

[Music]

he

[Music] w h [Music] w [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just tring something this okay I to I'm just TR to give you [Music] something I'm just tring give something I do I'm just TR to give you something [Music] w

[Music]

[Music] [Music] I'm just TR to give something I do I'm just TR to give [Music] something I'm just TR to something I I'm just trying to give you something [Music] h [Music]

[Music] w oh

[Music] [Music]

a [Music]

[Music] a [Music] [Applause]

[Music]

[Music] [Music]

a

[Music]

[Music]

[Music] n

[Music] oh [Music] oh

[Music]

[Music] [Music] [Music] [Applause] [Music] a [Music]

[Music]

[Music] [Music] [Music] [Applause] [Music]

[Music]

[Music] he [Music]

[Music]

[Applause] [Music] hey hey hey hey [Music] a [Applause] [Music]

he

[Music] n [Music]

[Music]

[Music] track [Music] hey hey he [Applause] [Music]

he hey hey hey hey hey [Applause] [Music]

he

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music] he [Music] w [Music]

he

[Music] h

[Music]

[Music] now [Music] [Applause] [Music] [Applause] [Music] oh

[Music] just [Music] something I'm just tring [Music] something I'm just something I do I'm just tring something [Music] a [Music] w

[Music]

[Music] [Music] I'm just I I'm just try to give you [Music] something I'm just trying to give you something I I'm just trying to give you something oh [Music] w

[Music] a

[Music]

[Music] [Music]

[Music]

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music] a

[Music]

[Music]

[Music]

[Music] oh [Music] n [Music] a

[Music] h

[Music] oh [Music]

[Music]

[Music] n [Music] [Music] [Music] [Applause] [Music]

[Music]

[Music]

[Music] [Music] [Music] [Applause] [Music] a [Music]

[Music]

[Music]

[Applause] he hey hey hey hey hey hey [Music] [Applause] [Music] aah

[Music]

he

[Music]

[Music]

[Music] track [Music] hey hey hey hey [Applause] [Music]

he hey hey hey hey hey [Applause] [Music]

[Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] he [Music] [Music]

[Music] e [Music]

[Music]

oh

[Music] h

[Music]

[Music] w oh [Music] [Applause] [Music] [Applause] [Music] I'm just try to I'm just trying to give you [Music] something I'm just trying to give you something I do you I'm just TR to give you something [Music] [Applause]

[Music]

[Music] [Music] I'm just TR to this okay I'm just TR to you something [Music] I'm just trying to give you something I do for you I'm just trying to give you something [Music] oh [Music]

[Music]

[Music] [Music]

[Music]

[Music]

[Music] [Applause]

[Music]

[Music] [Music]

[Applause]

d

[Music] e [Music] n [Music] oh [Music] n [Music]

[Music] [Music]

[Music]

[Music] a [Music]

[Music]

[Music] [Music]

[Music] St [Music] [Applause] [Music]

[Music]

[Music]

[Music] by

[Applause] [Music] hey hey hey hey hey [Music] [Applause] [Music] hope you had a good lunch uh we have a talk called the road to developers hearts for any of us to relate to if you're a software developer or a security engineer um and this will be presented by sing and beathy I hope I said your name right so today I'd like to thank our sponsors uh we have Adobe here we have blue cat Toyota uh three prism Cloud Sim Grim uh conductor one so without uh the help of our sponsors uh we wouldn't be able to put on such a show here so uh just a reminder to silence your cell phones and we'll have questions at

the end hopefully we have time this is just a a 20 to 30 minute talk so I'll hand it over to S thanks Katie welcome everyone hope you all are enjoying the event and uh staying hydrated and also thank you for coming to my talk speaking of my talk it is the road to developers heart a quick introduction to what I'm going to talk about for next 20 minutes and why am I talking about it I've been a software developer for a quite a long time now and I've been in the security space and building security products since 2017 I have had hands-on experience in both side of things during this journey I get to see how different the

perspectives are from both sides on how to solve a problem or what problems to solve even on top top of that every project comes with certain constraints it may be limited resources or time when you add those constraints to the difference in perspective sorry okay yeah thank you yeah when you add those constraints to the difference in perspective it creates challenges challenges become friction and things get escalated in this talk I'm going to talk about how to reduce those challenges and share some personal experiences towards the end we will have some Q&A and if time permits I would love to hear what worked for you in those similar situations before we go too far you

might wonder why is it important right if there is a security issue you can escalate push people to get things done and it works most of the time why does it matter how say security teams and software teams should collaborate in my opinion it is not about solving one or few problems at hand at the end of the day we need to protect our customer Safeguard their data and keep our system secure in order to do that software and security teams need to work as a same team to achieve a same goal that's why I'm I'm I'm talking about this topic I kind of split this talk into four Cate categories people process technology and cultural aspects I'm

going to start with the People based one which is building trust this is an obvious thing to say but it it is really hard to do consistently if you work together as a team day in and day out it might be relatively easy to build trust but most of the time software teams work with security teams on a short-term engagement there is not much time to get to know each other so security teams work with software teams for a short period of time mainly towards the end of the project and you identify some issues and you make the development feel like you are blocking their delivery this whole combination of things do not help build

trust so what can we do about it right unfortunately there is no one's standard checklist to build trust I'm going to talk about few things in the upcoming slid that may help build trust but at this point in time I'm going to talk about few crucial benefits of building trust early in my career I used to be very hesitant to share some of the things I thought might be wrong with my security counterparts mainly because I might get into trouble or it might backfire and things like that after working in security space for a while now that I know the teams are here to help and I trust them uh whenever I couldn't prioritize some of the nice to

have security things in my project I go talk to my security friends and I asked them to advocate for it it helped my case a lot of time so developers know the product in and out sometimes they know where exactly the issues are and what issues are in the backlog once you build the trust they will bring you some issues that you did not even know existed it is also a really good reconnaissance opportunity the next one is being available there was a time when it used to take years to sorry weeks to get any of the security consultation done if not months thankfully things have changed now by the time you finish the back and

forth communication sometime you even forget what the initial ask was or you might have moved on to different project project thankfully uh organization started to prioritizing security more now than before having easy access to security teams during project development or when fixing issues is extremely helpful than you think it reduces a lot of overhead cost and rework and r- revs it makes both software and security teams more efficient this is another thing that is very easy to say you know but everybody is busy and it's very hard to do in real life so find your Avenues it could be weekly officers are setting up a design reviews or getting involved with the teams development processes from the

initial stages or establishing a asynchronous communication Channel through which your response velocity is a little bit faster without impacting your productivity obviously so also staying engaged with the teams through uh of periodic security events or sending newsletters and letting them know that you are available to help is a great way to build trust and also it make them make you feel like you are part of development team as well all right I'm going to move on to couple of process based Things Early detection of issues comes with a lot of well-known benefits I'm not going to get into that preventative controls are even better but be cautious of Shifting to left especially in the product develop early in the product

development early in the stages developers want to move fast build something Scrappy and I trate on it if your organization have lot of processes or red tapes around you know project initiation or setting up the initial infrastructure it can be disruptive at the same time I'm not advocating for putting test environment on the internet there needs to be a balance it may require customizing some of your processes and that should still be within the limits of your organization's risk profile the other aspect of it is program management in the past uh one of the audit teams supposed to do a review of our software and they couldn't time until end of the year they started the

review in November they did a great job they gave was a bunch of issues to fix before they go on vacation and some of those issues were expected to be fixed by end of the year and obviously some of some of us are not happy things like that could have been avoided or at least a advanced notification might have been helpful we all understand things like log 4G happen but if something can be done in advance please do not let that be a surprise next I'm going to move on to a couple of autom Technology based aspects doing manual things over and over again is time consuming and it is annoying it would be really helpful to

identify or build some tools and integrate that with continuous integration process it's going to save a lot of time for both teams once I was part of a team that was heavily using database programming language for which there were no open- Source static code tools tool analysis tool available uh we were spending a lot of time manually reviewing things and eventually we found a tool we procured it and we integrated it and we all were super excited that it's going to work unfortunately the tool was borderline unusable mainly because of the false positives so we spent a lot of time uh fixing the false positives and eventually we made it work and it was it

was a lot of time we spent a lot of time initially but it was one of the greatest investment for software and security teams at that time so as a security practitioner you have visibility into a lot of security automation tools if you can bring that into development team's attention help them automate and also find tunate to reduce the false positives it would be a great win for both software and security teams

few years ago I was part of a security team um and there was a development team came to us for a consultation at the time we had a really good network security engineer in our team and he started and he was new to the organization as well and he started providing some Advanced Network solution that some of us here it for the first time and we did not have any prototype done or we didn't even know how that would fit into our ecosystem the teams left with more questions than answers ideas like that are great it pushes us to innovate but it might be a c good candidate for running as a separate project because it

is very hard for software teams to build a new security solution when they are still trying to solve the software problem so provide realistic recommendation that uh that you might have to consider some trade-off decision but uh it it is really meaningful and helpful to the development teams being specific most of the security questions can be answered with it depends uh it can be vague at times and uh honestly it can be annoying at times sometimes too so it is a instead it's a really worth the exercise getting into the specifics and understanding all the various options of aailable and you know providing a meaningful and specific uh solutions that will help a that will

help a lot um in terms of like in terms of the communication with the development team and then in terms of the efficiency of the teams as well you you could even do some lightweight tabletop exercise in the advance that teams can use as a framework throughout their development and then they can use that during their sdlc life cycle the next one is escalations most Organization for most organization security is job zero or job one however you see it if there is a security issue you will have to escalate and it need to be treated appropriately there is no question about that but when escalating things having a well-defined mitigation plan or success criteria again being

specific about those those things will not only help avoid lot of back and forth communication and Chaos it will also help hundreds or thousands of developer hours depends on the size of your organization the worst thing that can happen is I get an issue in the middle of the night and I try to look for an answer and if the answer is something like it depends this is not fun also you might be part of a larger organization with multiple security teams trying to run multiple initiatives or campaigns or smaller organization trying to do multiple things with good intentions in both cases software teams going to receive all the issues at once and if the team is resource constrainted

they're going to prioritize some issues versus others and some some of the issu is going to wait now the question become if it can wait now why not do it before right so ruthlessly prioritize security initiatives or campaigns within and across your organization it will not only help protect the security teams brand within the Builder Community it is also very essential to build trust across the Builder teams finally I want to end with couple of cultural things changes take time when you try to introduce new processes or practices you might feel like developers don't listen or like you know they are not um you know they don't care about security but most of the time that is not the case

because security was an afterthought traditionally now we are trying to bring that into mainstream now it's going to take some time and practice for teams to get used to it the one best thing you can do is besides doing your organization Security Programs find your allies within the development teams so they can act as your eyes ears and eyes and ears out there they can ask questions like hey have you considered security when doing things so reminders like that are very crucial for changing the security culture across the development teams I have seen organization change their security culture over the period of 1 to two years through constant reinforcement and reminders so changes do happen this is my final slide and my

favorite one until 2017 I learned about absolute minimum security that is required to do my job at that time I started working with one of the security engineer and for one of my projects and he taught me about lot of cool security things and he advised me to go to lot of conferences like bsides and take some trainings and I really enjoyed security side of things and eventually I ended up join their team that helped me expand my knowledge across various security areas ever since whichever the development team I am part of I always look for opportunity to do security improvements and advocate for security things and also I look for people with security interest and ask as

them to take similar trainings I took in the past so if it is not for that one security engineer I wouldn't be here and also I would not be doing any of these things so building security communities and mentoring people at work goes a long way you could potentially influence the entire organization through the people you mentor and Coach I think that is end of my talk and once again I really appreciate everyone for being here thank you thank you sing [Music]

he

[Music]

[Music]

[Music]

[Music] TR [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey hey [Applause] EX [Music]

[Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

he

[Music] h

[Music]

[Music] [Applause] w [Music] [Applause] [Music] something I I'm just to [Music] something I'm just I I'm just want to give you something [Music] w

[Music]

[Music] [Music] I'm just I'm just dring in [Music] something I'm just tring in something okay I do for you I'm just trying to give you something [Music] w

[Music]

[Music] n [Music] [Music]

[Music] he

[Music]

[Music] he

[Music]

[Music] [Music]

[Music]

he

[Music] this is discover the hidden vulnerability intelligence with in cisa's key catalog presented by Glenn Thorp and what's that yeah oh I was just agreeing and we'll take questions at the end it's a short talk uh 20 minutes so uh we'll have 10 minutes for questions and just want to thank our sponsors uh Adobe Toyota um we had uh number of them this year Sim Grim blue cat uh three prism Cloud uh if you want to take pictures just make sure that you're just getting the screen and nobody's in the background and uh again we'll take questions at the end all right Glen is the is my volume okay okay so um I'll start off thanks everyone for for coming

to my talk um as mentioned I only have about 20 minutes so I'm going to speed through pretty quick and then we can have questions afterwards or if time didn't allow we can have uh hallway conversations or whatever so um um a lot of there's a lot of uh visualizations on the on the presentation so afterwards there'll be a link to to grab all the data the visuals the slides everything so don't fret about the actual um slides themselves and don't fret if you don't get to kind of consume the whole the whole chart because there's a lot on some of these and I don't think you're going to do that in like the 10 20

seconds that it's on the screen but um I'll just jump in So today we're going to dive into the hidden vulnerability intelligence that lies within the the CIS Kev catalog um more specifically what you can do to prioritize when you have more than one competing bone in the Kev so in other words you know there's multiple things added to Kev and you have to pick you know between one or the other there are some findings um as we look through the data that can help you make that prioritization decision and to clarify sisa is not affiliated with this talk they have not approved it um I'm just a big fan so I hope that they appreciate it um they are

in the room so I'll be nice and also that means that if you have specific questions we can have them cornered and make them answer your questions so we'll get started uh first off I have to introduce myself my name is Glenn Thorp I work for grey noise intelligence uh I lead the security research team there and I have to start off by saying they're an insanely insanely talented group of people I'm very fortunate to uh to work where I do and do what I do with the group that I do do um I've been in security for over 21 years different Focus areas such as detection response vulnerability management and emerging threats um when

I'm not doing cyber security things I'm usually uh studying weather patterns or scuba diving observing sharks stuff like that that's my whole personality basically and uh yeah so just I'm underwater either literally or metaphorically so that's just how we that's how we do it in security um so we'll dive in so what exactly is the Kev catalog is anyone here not familiar with with the sis Kev catalog okay great no no problem there so um I knew there would be people here I was ready for it um so Kev stands for known exploited vulnerabilities and I'll start by saying note that it's past tense which means it's already happened um it's not a predictive tool it means these are known

exploited vulnerabilities like it's in the name it's pretty clear but I think that there there's some confusion around that where people think maybe sometimes it's a bit more um speculative than it really is it was launched in November 21 and to be included in the Kev catalog a vul has to have three major attributes one there must be a cve assigned makes sense uh the vul must be actively exploited in a way that's impactful so for example a misnomer that um I've picked up from watching some of uh the sis Todd talks on the Kev um kind of doing my research here is there's a there's a misnomer that if it's in the Kev that means it's attacked or affected

government institutions or organizations directly and that's not necessarily true right that's not necessarily true that just means it's being done in an impactful way it could be a you know a large organization or something with a large footprint or another country whatever but it does not imply necessarily that it has affected our government in the US um and the last attribute is there must be some kind of clear guidance on what to do about this problem otherwise you're kind of just kicking a beehive a bit um so that third one is kind of important to think about so the purpose the actual purpose of the Kev is to drive remediation and mitigation within the government so it it has directives

that back it the Kev has a um a due date within it and again so it's basically built by the government for the government but enjoyed by all for everyone to understand what's going on you know in the in our in our security landscape or our threat landscape but it's important to note that they are backed by some some regulations and directives for the government agencies um that are that are bound to them um so next question is has anyone analyz the Kev before and I say that in just because um it's been done a thousand times but don't worry I'm not going to bore you with like the the the details of the the same old boring stuff or the

same old like Topline top level findings except just a little bit because I have to paint the path paint the picture for the path that I went down to kind of unearth these like three main takeaways that I have uh today so bear with me at the beginning we're going to get somewhere towards the middle I promise so who is privileged enough to be on the Kev like what is the diversity of this list well um interestingly so right now there's about okay first off this data like I had to cut it off at some point so I cut it off the last week of June so anything that happened in the last two weeks not my problem um but so

there was 1100 entries across 180 vendors but five vendors account for half of all the entries within the cev catalog kind of kind of interesting so we'll we'll come back to that um vendors that are usually in Kev include uh a lot of widely widely used software like Microsoft Adobe Oracle open source projects like Apache or major security tool sets like uh or or uh security controls like foret or Cisco Juniper Etc um so there's a lot of diversity in it but it's interesting that most of it is just between those five vendors another interesting thing is that 77% of the vendors that are on the list were added within the first 12 months of the Kev being created so this

line kind of basically is the onee mark you can see you know this is the rate of new vendors being added so you know it kind of leveled off you know roughly after the first year the the rate of new vendors appearing slowed down quite a bit all right so may be interesting we'll see now is the CVSs distribution for these um for the vules that are on the Kev interesting not really um the higher CVSs scores are on the right the the height of the of the chart of the plot kind of indicates the quantity but really it's kind of as you expect higher CVSs scores mean Uno remote uh remotely exploitable and uh maybe no

authentication whatever so it's not surprising that it would lie here it's maybe more surprising that there's like a three and a half um CVSs score on the cev like that's kind of interesting but anyway so like I don't know so let's take a look at the trajectory of the Kev of basically how fast things have been added to it so again it was created late 2021 it goes up to end of Ju interestingly it's almost linear except for you see that Big Cliff there towards the beginning of 2021 or 2022 sorry does anyone have an idea what that might be from what maybe was I see not what maybe was happening globally that would cause a just a big

dump of new vules being added to the cev okay Ry you y the answer a war a war yeah the Russia the Russian invasion of Ukraine um so we'll come back to that so basically looking for all of these little tidbits of color that that were you know maybe not lying within the Cub that we could figure out like how can we get a little bit more intelligence out of this so the kevs basically updated average every five days like H don't don't read into that much um there's been I know once maybe twice I couldn't prove it that it was updated in the same day twice in the same day I don't know if anyone knows um

but basically like the you know there's they get added as they get added you know sometimes it's every day sometimes it's once a week whatever um so H again interesting but I don't know so we keep looking does anyone have an A guess for what the average age of a vul that's on the Kev is and don't get mad sisa I promise we're going somewhere so it's a thousand days so like if you just look at the actual average age holistically then the average age of a CV in the is over a thousand days and just like oh that's not feeling great weird and speaking of Ages I had to make this joke because it's too obvious but there is a

22-year-old B on the cev so that means it could play Taylor Swift 22 on repeat it could walk down Las Vegas uh or Paradise or whatever this it could walk down the strip and Gamble and drink um the vone could you know do everything that an adult can do in the US so pretty crazy but that does highlight like how integral it is for patch Management in organizations to be very thorough not just looking at what's current or what's new or what's just been released but also you have to keep checking your um your organization for these older things because they still exist we see them all the time at grey noise like people love

the old bones because if they're going to keep working people are going to keep exploiting them you got you know you got to patch it we all know that so so far this isn't really painting a very timely picture so how do we dig deeper we Dig Deeper by basically uh finding the signal and all of this noise and the good news is gray noise is all about reducing the noise so we need to accommodate for the outliers and so when I was looking through the data and digging for some way to make this more interesting than I think a lot of I mean no offense it's just like more interesting than what we've we've

already learned from the Kev like there's got to be something new here I was able to break down the data set into three different essential categories um the first category and they're time bounded by the way the first category would be the initial dump that was on November 3rd of 21 which is when the Kev was created so this plot shows like the width is the quantity the height is the is the age and so this is what the initial um dump of 280 something uh vules look like that were added to the Kev um from that age and and quantity perspective the next data set was that really large Cliff that we saw on that earlier graph

and so basically the invasion of Ukraine started on February 24th 2022 and the next 107 days there's no reason for 107 other than that's how the data kind of painted itself like as I looked at um the age of the CVS that were added or the Technologies or the rate how quickly they were being uh kind of batched together there was just a natural bookend for the 100 7th day afterwards so we went with that so basically I broke that down into what's referenced um for the rest of the talk as ukr conflict so this is that dump um where like I said when you saw and I'll bring it up here again in a second that

large Cliff um so you can see a much bigger diversity as far as age and that's where you get that 20 something year old vul at the top um and it's just really much more scattered and then the last category is everything else so when it really gets on that Center column Center column being basically you know less than a week of of cve age um you know everything kind of comes back together so what we're looking at here is three groups the initial one like I said 287 cves 591 day average the second one has almost a 1900 day average so that's where you get that average of a of a thousand days on Kev like very

misleading it's a thousand days because this one uh 100 and say 107 day period just really skewed it so you have to take that into account and then everything else and so interestingly enough the average age of the first dump and the and and the everything else category is essentially you know pretty close together so that's cool and then clearly the outlier is the uh Russia Ukraine Invasion dump okay same data different view um maybe it helps you consume it better some people don't like the violin plots and stuff so um but left to right is the age uh and and top to bottom is the quantity but same information and again this will be all

available for you afterwards so let's look at that cumulative view again like I teased um we put lines on here I hope you can see them but um basically you know marking off the beginning and the 107th day of the Ukraine and you can kind of see how that data really kind of level like literally levels out and then keeps its continued craw so the rate of addition is pretty linear which is kind of interesting um what does that mean I don't know we'll see but it's just interesting um but what we do know is that there's the first like full calendar year of um of the Kev that doesn't have like a major outlier is

2023 so we're kind of just going to say 2023 is really the Baseline because 21 was only a month of data and and 2022 was just so heavily skewed by the by The Invasion let's check back in on the cbss scores with these data sets broken up does that look interesting I think not um they're almost all NE identical especially the top and the bottom are really really close together um and even the CVSs score of the conflict group is pretty close together so um I think I'm kind of done digging into the CVS score looking for some some interesting things there because it's it's pretty pretty well represented uh across all three data sets so okay so those are kind of

the basics right like we're going to step it up a notch so question of course it's a trick question I'm giving a presentation about it do you think the average age of TV e is increasing or decreasing over time are they getting essentially like added to the Kev closer to their um existent or their their known about date or or later does that mean sooner or decreasing okay good answer so yes they are absolutely decreasing so this shows it broken down by year 21 2 3 4 um further to the left on each one is basically the age of the cve and then again height is the quantity so again 20 the first year little abnormal because it was the dump

second year the Ukraine war like everything's crazy but then once you get to the you know quote First Baseline year of 23 you can see that the cve age is actually quite young like within the first week of their of their age um of their uh assignment and the trend continues into 2024 so so now it's looking a little bit better um a couple of thoughts on this I think the the age like the age of cve being younger is part well part of it's definitely due to attackers exploting things faster because they are we know this like that's not news but I think it also speaks to a bit of you know CIS has been

very serious about doing Outreach and Partnerships and info sharing and stepping that up each each year and building relationships that's why they're here um I think that's why you're here so it's it's you know I think I think it's paying off because people are sharing more earlier um and openly and so that that helps us all it it helps everyone so and there's the ability to submit to the Cod online soon soon so they're working on it so it's great so 2023 is the first Baseline year like you like I said you see the major shift to very early uh or very young BS being added so Kev is again looking a bit more timely once we start taking out

those anomalies another question is the Kev data static and this one I think might be surprising to some folks so I'll tell you that it's not static and I would ask other than like the like there is an occasional removal of a vul from Kev don't worry about that but within the Kev is there a fe what field might be updated near silently I think it silently So within the Kev I should have brought up the actual fields in the data set at the beginning but there's a field in there that is known ransomware campaign use and so this was added last October so it hasn't even been in a full year and it is to do what

the name says is this known to be used in a ransomware campaign and so the options are Known Unknown that's it right is there one more yeah known or unkknown so I started digging into this and oh well first off we know that field matters because there is data to suggest it's not our data but um there is research data out there that suggests that when a vul has the known ransomware campaign used attribute it's patched two and a half times faster that makes sense um ransomware is expensive we all are very familiar with it so you know okay cool so this is a five minutes for okay oh my goodness I am behind all

right we're gonna fly so yes so basically there's a known Campa ransomware campaign Ed flag it does get updated silently and so what this looks like is again offline uh consumption but the 40 there's been 41 times where that field has been changed um after being added to the data set and we found this by basically harvesting the the Kev every day and then doing a dip to see when something changed so as far as I know this isn't really publicly announced or uh displayed in some way so if your organization cares or utilizes this field in some way then it's important that you go back and check on it to see if it's changed it is

a one-way street from unknown to known it's never reversed again makes sense um but for the secret Intel so that's just kind of an interesting thing in case you didn't realize that um but how many people or how many organizations pay attention to the time to fix the Vol so again on the Kev is a due date that due date minus the day that it was added is the time to fix it so a lot of folks probably don't because it's basically meant for government organizations however it can be telling when the Kev started we had basically a default of 180 days or uh 14 days and then about the time of the Russia Ukraine war you

can see it standardized on 21 days and that has continued but as we get down uh to the bottom right there you'll see as of late there's been more additions um to the KB that have a shorter time to fix so that's really going to show you uh some insight into kind of the level of concern is what I'm calling it that is uh that is known about this vul based on either what they've seen or maybe the uh the threat landscape whatever and so the last deep find I think is what matters is the day of the week that something is added to the Kev I think this is super cool um basically again left to right is yours so early on

kind of all over the place we're just figuring it out standardizes in 23 24 continues except it gets really quiet down there on Fridays and those Fridays had had a time to fix of seven days one of those had um oh yeah time to fix of seven days one was a foret one was a Palo Alto pan phone so essentially when you dig through this and you're having to look through the data to figure out like I need I got more than one thing to fix on the cev what should I prioritize basically dig into what day of the week it was added that's an interesting tell um what is the time to fix on that bone that's an interesting

tell um and then lastly the the ransomware campaign use again if that's important to your organization your processes you definitely want to kind of check back on that and maybe like we can work with sisa to like flag that when it gets changed um so I gotta I gotta wrap up I'm somehow far behind um and oh yeah don't try to predict the cev like it's just not a thing but if you have to look at vendors that are already on it because they got added you know that 77% number and then of course like attack Vector none user interaction none privileges re uh privileges required none is a good start but anyway so here's uh my information this link will

have the slide the data um how to contact me um shout out to Bob rutas he's the one that did the visuals for this you may have seen his work you probably have you just maybe not you don't know it um he's amazing he's the goat for day to work um and shout out to feedle I don't know if anyone from feedle is here but um they don't know about this but um we're a customer and they just when you do a good product and you make jobs easier I'm going to call you out for it and so they have so hopefully they'll give us a discount next year and that's it that's it thank you Glenn we have time for a

couple questions okay great any questions okay so so if no questions um can get with Glenn um yeah after this show thanks

[Music]

[Music] h a [Music]

a

[Music] w [Music]

[Music] n [Music] [Music] [Music] a [Music] [Applause] [Music]

[Music]

[Music] oh [Music]

[Music] a [Music] [Music] [Music] [Applause] [Music]

[Music]

[Music]

[Music] our next talk is with odjob and the title is nothing went to plan because you didn't have a plan so I just want to quickly thank our sponsors Toyota Adobe blue cat Sim grimp uh conductor one uh this talk will uh will hold questions till the end and then I can pass the microphone around so we can hear them because we're recording and this is streaming live okay so oh we're live we are live are and we are recording fantastic cool so that's always fun things said live being recorded are always a way to make plenty of mistakes that you can't take back all right so imagine that you're a ciso and one day you wake up to news let's

say on a Monday because all the fun things happen on a weekend right uh you wake up news to news Monday that your company was breached and you check your emails and find that teams were busy without you at work um in an email thread legal forwarded to you the night prior you could see everyone making decisions arguing taking bold actions and the uh and the chief of operations decided to make a statement to the press that statement to the Press uh apparently uh was filled with inaccuracies it gives way more information that should have been released your security team also seem to be unaware of the matter and just received emails themselves about this

matter and now the CEO is calling you and wanting an update for the board and your audit Department are Fielding questions for third parties that that need your attention welcome to a Monday but everything went to plan or rather things didn't go to plan CU you didn't have a plan and uh things never go to plan when you don't have a plan so that's uh that's kind of the name the the name of this talk and how we how we came here but they didn't have a plan so we're going to make one now so um basically uh I'm ajob and uh my laptop really wants you to know that it is disconnected from Wi-Fi

um but I'm oddjob I've been hacking for 12 years uh I'm a Noob uh my uh my first Defcon was uh 20 uh every time I hear someone say like oh yeah Defcon 6 it's like Noob someone's always in the room was at an earlier Defcon than you um I you might remember me from such talks at bsides Las Vegas as uh busting biases and infos seex so that was a was a fun talk I think uh from a couple years ago it's uh also streamed and also recorded um I also like to brew me um so if you have questions about that happy to share that hobby I also host the uh a YouTube channel called a glass of zero J

I basically just talk about interesting things I have probably stayed away from the crowd strike stuff because we're still finding all sorts of fun nuggets of information and things that we got wrong and I like to have information correct before I just go ahead and uh say a bunch of stuff about it but uh need to get on that soon uh and I'm also the senior director uh of detective response for a health company um but I'm not here in that capacity so even if you know what that company is uh I'm here in my own personal capacity uh but that is kind of more of a level of my experience and uh where I've been in uh different

companies uh throughout my career so far um so you know when we need to think about this as uh by the way enjoy the lovely AI art um a lot of interesting things totally not related to this talk but totally interesting things while I was creating like little bits of AI like oh yeah ceso wondering uh you know how where to start on building a cyber incident response plan that that hand is cursed That Elbow is cursed um but uh every time you ask for just a person uh it always gives you a white man uh so then I ask for someone androgynous and they still always give you someone white uh when you ask for a non-white person

they always give you a black person so never of anyone of any other um uh potential ethnicity uh lot of interesting things with AI image generation uh but these are the cursed AI uh images that I came up with uh that one also uh how many fingers are merged into his skull there um which is actually an accurate depiction of what that ceso was probably doing at that time merging the fingers into his brain um uh so basically uh you know where to start with a cyber incident response plan right uh you actually may find out that uh you already have one uh your pred cessor uh even if you're not a ceso you may just be an incident response uh

lead or director um you may already have one uh that you didn't write someone else did and maybe they didn't do the best job at it or maybe what they did is it's 80 pages long and is you know they took niss they took cisa they took 12 other standards or other things they could possibly do and just crunched them all into one thing they said that looks good that looks good let's put it all into one document that is not the way to write a Ser at all and you may think oh it just needs to pass compliance if your compliance people are worth their salt they're going to say yeah this is not good this there's no

way anybody can follow this and know how to work with this um but that's that's what I find with a lot of people's cyberin response plans way too long way too many details not the right details um usually too many details about the wrong things you need to actually be worried about um but hey it's a place to start uh that may already have approvals by your compliance department or quality depending on what all you need to uh get approvals for um so that may be a place to start you may decide that I can chuck all of this away you may actually look at some of the roles and responsibilities and figure out oh

there's some people here that may need to be involved in incidents I wasn't even aware of so kind of do some reconnaissance uh if you do have a cyber incident response plan that's poorly written or maybe suboptimally and start there and see what you can salvage but what we need to do when we start with a cyber incident response um plan is we need to start with people so I like the framework people processess technology does everybody know has everybody heard people process technology all right okay so it's a way of thinking um really I was kind of thinking about this earlier today even like just people process technology that's the way in which you should really set up your

organizations you should look for good people you should put give them good processes to follow and tools that you give them should be able to align to those processes I see so many people start Information Security Programs with we're going to get tools hey and then we're going to we're going to figure out how to use those tools and then we need to hire people who know how to use those tools so then you get job descriptions that say I need someone with like 15 years of this particular tool not just firewalls in general foret specifically uh even though foret may not have 20 years of existence out there or maybe their nsse licenses haven't been out there forever

um you still have to have that and so that's where you get these job descriptions that are just overloaded and overburden with tool specific things that you don't need um but that's where you start with tech and go backwards uh think about it you know Tech really Drive allows your processes to work your processes allow your people to work so you can really start from anything and understand how each part processes the other but really we need to think about the who we need to think about our people um and not just our people people outside of our organization as well because we have to report to people as well outside of our organization in a in

an incident especially nowadays with the SEC uh knocking on people's doors wanting to know uh about that breach that you should have reported on your uh what is it 8K or 10K um so uh one of the first things I like to to go and do is get your key contacts um so getting to understand your folks like your technology now I put technology smes there first because I think it's the most relatable to us they're probably not the first people you even need to go talk to you may actually want to talk to some other things and and figure out some other people like your lines of business in product and Manufacturing and sales uh

you may not have OT or skada in your environment um but I'm just putting it in there just to kind of help you understand where these folks are um customer success and service can be very important to understand where they are um your business support in HR right so if we're doing an incident that involves an Insider threat do you have authorization to go prying into their personal lives and go prying into the data that may or may not be their browsing history and start uncovering things that they may need to be doing after all an Insider could be taking information from the company and putting it into their own Google drive or proton mail or something else um do you have

authorization do you know you have authorization to go ahead and start seeing what are they putting into these personal things um maybe they are even using platforms such as uh Health platforms to uh um have notes because usually Health platforms for instance have an err of privacy around them you actually don't want to uh start decrypting um you know Network traffic out to health platforms because that's someone's Phi and now you might have to be responsible for that um so you have to probably involve HR to understand at least what's your kind of your standing orders what can you do what are clear clear engagement points you can do or criteria to do that and where do you

need to say hm before we go further let's make sure HR is clued in and you know what let's throw legal in there too because we're probably dealing with laws especially if you're a global company uh how you handle a German citizen is going to be very different than the way you handle American citizens how you handle a Chinese citizen is going to be very different than how you handle uh an Indian citizen they all have very different privacy expectations and especially where that data is allowed to go you may have to keep all of your evidence or data if it's about people in a particular location so those are important things to kind of keep in mind

again these are great people to have you don't know even what technology what process to put into until you start talking to these people that's why people are so important let's talk Finance right so in the middle of an incident you think at some point you're going to need to do probably one of two things you're going to need to either tell your insurance about something your finance department usually handles insurance so you're probably going to need to partner first of all probably go talk to them before an incident and ask them about insurance but you also maybe have an incident response retainer right and so how do you pay for that most incident response

retainers are pretty good and that you know they don't want money up front uh they they they figure that out out after the fact but you're going to still need to make sure Finance is in the loop so that they understand what really is going on here what's the contract here what's the velocity of the cost you know are we talking you know we have to do 40 hours at minimum at $300 an hour what are we talking here and how much is this going to cost and do we want to actually engage in that cost right um they're going to help make that decision facilities facilities and physical security can be very interesting what happens if you're in a

situation uh where you have for whatever reason abandoned an office uh evacuated it but you still need to go in someone needs to go in and be able to pull out hard drives uh and do forensic examination on those how do you get in and access to these places to get your hands on things or get somebody's hands on them right so those are people you'll probably want regulatory in quality pretty obvious there uh you've got laws to comply with you have laws and organizations that are going to be coming after you wanting to know about things you need to know what clocks exist and when those clocks start ticking again legal reads these contracts and laws as well they're going

to help you with these definitions sometimes legal well hopefully most of the time legal's the one making the decision when a clock starts for your company um but keep in mind uh and talk talk to your Le legal team about what does the contract language say for instance does it say uh within 24 hours of a suspected compromise of data you are to notify me as the customer what's a suspected compromise every similer in my experience is a suspected compromise it's being brought up for a reason we have an alert in that for some reason it could be false positive but right now it is a suspect event it is a suspected incident at any time so you know they

may uh you know they may not be so happy to be uh getting subscription uh or uh notifications from your sim every day hey we we got an incident we got a we got a we got an event we got an event well that's probably not what they mean they probably mean something else confirmed compromise or confirmed incident is language that I know I prefer and some other folks I've worked with prefer because that now is this is the time we can document it this is the time we actually have confirmed there is impact that is impact that is defined as to the parameters of these contracts these laws these regulations so pay very close attention to that of course your

legal team should be but this is always a good talking point because someone may look at that and not know that you know some of these things and they may go like yeah sure suspected compromise what does that mean you may have to help educate them and help them understand that and of course your technology Ms uh smmes I kind of went in reverse order there um you are not the expert in everything right you may have come from database you may have come from application you may have come from uh identity access management or even privacy but you do not know all of those things uh even if you're a database person you may have just been a pro

postgress database person have you ever touched Oracle there are interesting nuances there hey who all what all uh queries did uh you know user a make on Oracle database from this time to this time may take you a while sure you can Google may take you a while or you need to know who your technology smmes are to be able to contact in the middle of an incident right and get get people especially in technology but in some of these other areas too get a primary and a secondary because guess what some people are on PTO and in some countries depending on if yourm is in uh like I said Germany or in Europe uh PTO is sacred you don't

come off PTO uh you you you're on vacation and you don't have to answer your phone or anything so who else is going to take that call hopefully right and then hopefully they're not on vacation at the same time uh but have a process for escalation to get somebody else who can direct you to anotherme right um and really all of these people don't need to be involved in the same types of incidents right and so what we really do have is we have you could possibly break sir sock and CT out to two groups I know people who do that but really two groups exist here you've got the people who are more boots on the

ground handling the incident dayto day so this is your security operations center these are also your extended teams more in a not executive leadership capacity but in a leadership capacity in their various lines or departments um and so they're going to make decisions like what does a communication to our customers need to look like what language should we avoid hey we're going to make a statement to the Press first of all why but we're going to make a statement to the Press is everybody good with what's being said here um and so you know if you're a ceso or you have a ciso they're going to want to definitely be on on that uh you know customer success is

probably going to want to be on it if you have corporate comms or external comms you may even be engaging an external party uh such as uh crisis Communications and things like that they may come in and say we recommend you make a statement like this stay away from this language um so you you'll definitely want the those types of things but but generally that's what I mean when I say sock insert and the sock of course in involves your uh your your usual suspects your tier one through three your incident response folks uh your Security Management and also it management and uh the smmes there as well um again kind of those usable suspects um technical tasks you're

trying to eradicate the threat and restore uh restore the functions of the business um executive committee okay this is the O crap moment this company could be gone tomorrow uh or maybe it's not that bad but it still is something that requires uh you know the CEO or that person's direct uh um reports attention and so uh you know so we we talk about making decisions at a more strategic level about how to handle things um a lot of times uh hopefully uh whether or not to pay a ransom for instance um is probably well above your pay grade uh hopefully is uh and is more so at a board level or at an executive

leadership team level so uh definitely those people need to be involved at that point um but it's very important to kind of keep these roles separate there's usually for every line of business there's somebody up at that level who has ultimate ownership of that area so it's Chief legal counsel you may have legal who you deal with data day in your sock CT but you also have the chief legal counsel um so that person can be different or maybe they're the same right so it depends on your your company and uh what they're doing I want to make sure I have all my notes here oh good fantastic so there's uh there there's a little bit of an example and by the way

there will be a QR code at the end to scan uh that will take you first of all to our lovely friend Rick Roll but also uh will take you to my gitlab where you can see the slides as well as an example serp to get you started um don't just copy paste uh There Are Places to replace your company name but also other places to uh uh fill in more information but you're going to see like this roles and responsibilities you're going to see titles you're going to see names some of the titles are more functions within an incident like incident leader that's not a maybe it's a Le T maybe incident response leader is a uh title at your

company but in this case it's more of the who's actually running this incident it could be different people but it is a role nonetheless for a given incident that needs to be fulfilled um legal ethics and compliance right that that probably is going to be assigned to a particular person in that situation but anyway this is in the document I just want to show you what that what that generally looks like you'll have responsibilities and of course how to get a hold of those people um we come to the process part right so people we now have our people understood now we need to start laying out process and kind of when you start figuring out process you start doing

boring things like talking about scope and definitions so let's talk about definitions um one of the first definitions you're probably going to want to craft is what is a security event what is a security incident what's a breach what's a compromise those are very important things especially the B- word breach is a very important word uh there's a lot of contractual things that talk about breach there's a lot of regulatory and legal things that talk about breach consult very carefully with your data privacy Regulatory and legal folks they are going to probably tell you what breach means you probably get some input on it but your opinion does not matter as much as theirs in this

Arena but very important though is what is the difference between a security event and a security incident to me incident is where we start looking at confirmation we have a confirmed event here a security event could be anything a security event could be a third party just told us that there's comp credentials on the dark web okay can we verify that um it could be uh you know somebody you found uh an event where Microsoft told you someone clicked on a fishing link well was that link actually fishing I've seen false positives right so how do you go and and take care of that so everything coming into your sim every time someone clicks on that fish alert

button every time someone says Hey something funky just happened on my laptop I think it may be compromised those are security events we then go through a process that we'll talk about in a little bit to determine is it actually an incident or is this and how bad is it so incident severity this is where we already start kind of telling uh the boundaries of what an incident is um this is very high level there's more details about incident severity in the example serp it is suggestions uh you can put more definition in there you can even add a fourth severity and I've seen some people put a fifth severity level in there three is probably a good minimum

you have a high medium or low right um I like to have a fourth one is critical two is high and the real difference between Sev 2 and S one to me in in in serps I like to do is am I getting the CEO out of bed on a 2 a.m. 2 a.m. on a weekend that's a s one if I'm not it's probably a Sev 2 we might reevaluate and see if it's a Sev 1 later right so that's generally my little gut check it's not written in the in the Ser that way hey if I'm waking up the Caso at 2 am on a on a Saturday doesn't get written that way but that's the idea

that's the feeling behind it um but here is just generally right we have low impact right and impact can be anything impact could be Financial right maybe you had a business email compromise and someone walked away with 3,000 if you're a multi-billion dollar company $3,000 you lose track of that you know every every hour you lose track of $3,000 every hour every transaction you don't care probably it may be a low severity impact based purely on the financial aspect but if it's maybe a $5 million loss because of some transaction that was going down someone got in the middle of then that may go towards a high impact right so there's materiality to look at too uh as far as impact and

we are finding out with some of these lawsuits that are going on um we don't really know what the definition of materiality is with the SEC uh so the jury uh literally is still out on that um and uh we're still figuring that out but uh definitely talk to your lawyers on what they think materiality means for your company uh because again they get to say on that you're you have input but their opinion matters yours doesn't on that matter so go with what they say um you'll also be more protected in that way too it's like you're at the advice of council I defined it this way um we'll talk about privilege in a bit too also kind of

think about severities who's getting involved at a severity 3 there's no reason to get the CEO even involved there's probably not even a reason to call your Chief technology officer up or you know someone else your CFO there's no reason probably to bother them most likely except for maybe after the fact maybe on a quarterly basis you say here's the incidents we had in a summary of those um so severity three I add that fourth level on there for kind of your everyday things I like adding a fourth one to say hey malare got installed on this person's machine it didn't do anything we caught it in time it was prevented we've cleaned the machine or

maybe we've even got a laptop uh swap out going on we're good confirmed that yes an event happened but nothing of impact happened to us and we prevented further you know thing bad things from happening I don't need legal involved in that unless there's some I don't know maybe boundary that was crossed as far as what device it was what data was potentially on there I don't really have to go further into that right um so you may want to add that fourth one in there otherwise severity 3 is pretty severity 3 2 and one give you pretty good ideas on how to delineate um you also have incident categories right and these incident categories do a lot of things they give

you tags for instance to start figuring out hey how many uh malware events did I have this year how many uh inside threat events did we happen oh by the way Insider threat um you may be thinking disgruntled employee who's going to sell data to another place maybe some corporate Espionage maybe you hired North Korea and didn't realize it um no before if that's not a publicity stunt um and so uh you know you have DOS uh but Insider threat also is people who click fish links uh people who accidentally download the wrong file Zilla if there's a right file Zilla even um but uh uh but people who download those types of things um that is an

Insider threat user error user mistakes uh that is what's considered Insider threat it does not necessarily call malicious intent right okay um you know compromise credentials fishing a lot of these things can be the same incident you know you could have a Mau incident that involves compromised credentials as well you could have a zero day exploit that also involves business email compromise it allows you to understand the complexity and the multi multifaceted nature of a particular incident I've even seen people use categories along with the miter attack uh framework to kind of figure out at what point or kind of where did the attack or get to if your incident is purely more on a a

reconnaissance level or maybe initial access level and nothing further that may be good data to Keon later right in an incident okay we had a lot of initial access events and we're stopping those from getting further uh oh we had someone get lateral movement how do we make sure that we don't get lateral movement to these people in the future you don't do that unless you categorize your incidents so being being able to not only determine if it's an incident assign its severity and get the right people involved and make sure you have categories so that after the fact and even during the fact you understand what is the nature of this uh incident very

quickly helps a lot um we also have as part of our process we have incident phases right

um yeah Okay cool so yeah so we have incident phases first of all is identifying right we have identifying incidents uh this is anything this could be third parties talking to you you know your sim alerts uh we have triage and escalation this is where you really start saying what is the severity of this incident and then depending on that severity who do I need to call to start actually an incident Bridge right so does an incident Bridge even need to be started and who needs to be on it um that that's where your sock is going to start realizing very quickly how how big this is and get the right people involved um and you know again this is

also where you know legal might say okay we need to start preserving evidence we're going to start uh privilege um that's very important to make sure legal can establish privilege you don't get to establish privilege you're not a lawyer uh this is attorney client privilege right things that are said to your attorney as far as the matter is concerned and the advice that they give you is not something that's generally subpoena without certain exceptions I'm sure especially if it's of a criminal nature uh that generally well to cover up a crime uh for instance uh would possibly not uh go for um uh go for privilege but I'm not a lawyer um but uh you know privilege is very important you

can't just invoke privilege usually you can't just invoke privilege by copying your lawyer in there that's just not how that works they they are the ones to invoke privilege and give instruction that this is to remain only with these parties if you talk to someone else do not give them information unless by a need to know basis only again at the direction of council uh contain and erad um containment and eradication so now we're trying to stop the bad people we're trying to kick them out of our environment as well right so this is also where you've probably called in your incident response retainers depending on how bad this is uh you've maybe isolated people's machines reset

their credentials kick people out of their accounts for a time being um and that's where that starts happening uh you have your restoration and Recovery pretty much what it sounds like we're getting the business back up and going we're getting users access sometimes it may have to be crappy access maybe they had a nice laptop with everything they needed now they're going to have to deal with a vdi for a while uh until you can get them a laptop so they may have degraded um uh degraded experience but experience nonetheless and recovery so you may also notice that you aren't really responsible usually for recovering data right you are not the B BCP nor are you the Dr uh specialist but

those people are also a part of your incident response um probably the most important step in this entire process I would say is your postmortem and uh your your post incident activities these are this is where you learn every step of the way where did we go wrong what could have been better what things did go well what saved our bacon here from this being even worse situation very important to call out especially to Executive leadership uh you know imagine your board saying yeah we had these severity 2 or severity one incidents here's the thing though because you invested in these things you got our people involved uh and paid for you got you know the right resources we're now

we were able to uh keep this from becoming an even worse uh a worse matter so your investment is working but maybe a change in strategy maybe moving this PRI this strategic item up in priority uh may help us in the future to avoid this situation also do you need to uh improve your Ser uh hey we didn't understand really if this was a Sev 2 or Sev 3 so we didn't know what to do so we just called a Sev 3 what we should have called a Sev two what do you do in those situations well you might want to go back in the ser and you might want to add some clarifications or some hey you

can also kind of uh escalate it a little beyond what it probably is to get to get initial action urgently going and then you can back off later right so those are different types of things you might have in your uh postmortem um along with each phase of these right you're going to have very specific tasks and again this is in the serp that's in the example at the end with the QR code um you basically have what phase we're in you're going to have the type of activity that is going to be carried out and then who is going to carry that out so this is a very important thing where an executive for

instance should be able to look at your Ser should be able to see okay where am I in this oh okay who where are my peers involved in this legal other people who makes the decision about this okay I could see that and then where do I make decisions oh there's me there's me there's me and now I understand what I'm here to do where are we oh we're here down in the escalation phase okay we've We've Just Begun basically fantastic now we understand where we are you know it's a big old map that says you are here um you should be able to do that within like 10 minutes of reading a Ser understand those things and if it's 60

to 80 pages long that that's not going to happen so you got to make sure we keep this brief lastly we have the technology part of our incident response so there's all kinds of areas here right um we've got things like how are we going to even track this incident so we've got a nice we've got a nice Ser it's a great process but how are we going to document and how are we going to keep these things together yes even saying I'm going to type this up in a one note notebook or I'm going to type this up and put it into snow or jira or a super secret SharePoint that only some people have access to it's very

important to say here's where information about this goes here's where we're tracking this here's where we're tracking po incident activities very important to keep that documented and understood um and again understanding you know you may even need external incident response platforms because what happens when all of a sudden uh uh particular kernel uh level uh software ends up taking out your entire business um what happens with something like that or a airplane crashes into your data center you know what happens when something happens that you know really disables your ability to do business even the business of security you may even need a third- party system to track and work that incident even bringing in

those Communications which kind of brings me to out of band Communications um let's say you can't trust talking to your CEO on teams or your Chief legal on teams or even your ceso or your team members on teams maybe you see that somebody now has 0365 admin they can read everything they can get into everything and see it so now you need to go someplace they don't have access to that is already pre-arranged and understood and the and the best time the worst time to decide to use out ofand Communications and what you're going to do is during an incident okay how are we going to how are we going to do this um quick and dirty many times people will

go signal signal is a great way to do that well yes and no first of all you already need to know that we're going to use signal and here's how we're going to do maybe we have a maybe you even have a super secret Treehouse club uh you know a password and a challenge response phrase uh that you're going to use but you have to have something already pre-arranged so that people understand what you're trying to do even within your own sock team um but you might need to do that just initially to uh get some things going um the problem with signal is it is the beauty of signal is it's endtoend encrypted signal doesn't have a

copy of any of these things the bad thing about signal is it's endtoend encrypted and your device now becomes toenable if a lawsuit is involved potentially again I'm not a lawyer talk to your lawyer about this strategy um in a pinch it could probably work for some things to at least maybe get some things going but I wouldn't want to discuss too much on Signal um I want to go to maybe a Google workspace right if I'm a Microsoft shop or if I'm a Google shop maybe a Microsoft space completely separated from SSO and I can whip up these accounts really quick and get people involved maybe through their signal um I can get them into their

environments we can do we can do our own call bridging over there there are also platforms that literally just are geared towards being an incident response platform where you can bring people in securely and you can also then have communications uh that are that are encrypted secure uh and The subpoena goes to that right you don't have it on your phone where you may have content and other things you don't want courts uh potentially getting access to uh even if it's just a market as not responsive um so forensics uh who in here uh has a forensics arm of their company oh one cool awesome it's not something now now you all probably do a

little bit of forensics you know figuring out what artifacts are going on or how did this attack happen on this machine but you're probably not whipping out end case every day you're probably not doing a lot of other you know memory memory analysis whipping out volatility every other day to do this you're going to need to bring in the experts it is such a ridiculously specialized field out there not only that but how many people in this room can give expert testimony in court and are certified to do that yeah no that's your incident response retainer they have people like that on staff who who literally are are are built to do that and can say yes I

know that image of the hard drive is the is the same copy unmodified because of this and yes they're going to use check sums and other things but they're going to be able to give that witness test witness witness testimony hopefully not with hard drives witness testimony to it uh you can't do that um all right I'm going to make sure on my on my little laptop it kind of kind of goes crooked so I have to kind of go like this one um all right cool so wrap it up already we've been talking a long time about incident response plan um all the things you need to go in I think there's uh there's a basic process to

writing this here and I think you can get this in five pretty easy steps figure out your roles and responsibilities who your key contacts are um also have a little conversation with them you're not going to do the road show yet have a conversation especially with those more key contacts like legal HR finance and even get their insight into hey let's say tabletop a couple things with them real quick say we had this say would have beat business email compromise and this happened when do you want to be notified about that when do you think you need to get involved they may be wrong uh but they're more than likely going to be right uh in what what they think um so

have those little conversations to help understand where where in the process you need to accommodate them get your definitions right make sure everybody's on the same page with it right understanding when are we actually in a breach or not uh too many times someone will just use the word breach very half-hazard with a customer or a regulator those ears perk up especially if they're compliance or audit people and go well you didn't notify us well you use the wrong word you did that wasn't a breach that was just a malware event that uh was quickly remediated um get uh get the incident process laid out you may you may use nist maybe you're a nist organization and you have to use

nist completely fine to do so there may be other Frameworks you want to use this is a suggestion it is malleable I did take nist and just tweak it to my own little uh Hearts content uh because there are a couple little items I'm like ah you can merge those two um to make it a little easier Road show this during an incident is not the first time these people should know what the incident response plan is you need to Road show it and say you're part of this plan and oh by the way you're also the person who's doing business continuity planning you are doing business continuity planning right um you know you need to

have your own process because guess what this Ser is not your business continuity plan it may trigger and engage those plans but this isn't your your BCP and then test it right let's do unit testing on this uh let's uh in user acceptance testing let's do a tabletop let's get the execs down let's get the the technical folks down get your technical folks on a separate tabletop have them run through it this is interesting and do it before the ex excutive tabletop why cuz now you're got to find all the tools and capabilities you think you theoretically have at your disposal oh but actually yeah you you have no idea how to use them or do that oh hi this

database goes out what do we do well restore from backup yeah we haven't backed that up in three years okay cool write it down execs hey we do that well restore from backup yeah unfortunately that hasn't been backed up in 3 years there may be some surprises but it's very important to tabletop these things so that people start understanding and realizing what are the capabilities we have what are we missing what are our gaps and how do we need to prepare more for an incident kind of a a phase of cyber incident response planning I left off was that preparation phase it's a big cycle right we're always preparing um yeah and do this annually do it every

year bring it you can bring it internally but you can also bring in external a lot of times your uh incident response retainers have hours left over or have money left over at the end of the year some people use that for their pent test sometime just go ahead and say let's do a tabletop this year instead let let's pay for a pen test separately but let's use this for a let's use this for a tabletop this year let's get someone else on our pent test let's have let's have someone else have a crack at us um the very important thing been when writing is don't get in the Weeds on Tech uh this is much more about business

process um so if you if you have screen captures of of security tools or Network Tools or firewall rules and a whole bunch of other things uh you're doing it wrong uh there may be some appendices that that might go into those are probably more um uh uh standard operating procedures or standard work instructions that you want as playbooks that might uh that might follow on to a serp right um also like I said before executive should be able to read this in 10 minutes and also you should be able to to read it fairly quickly and kind of skim it and understand what's going on um again stay high level you're going to be able to

keep 20 to 25 Pages maximum and that's including the title page and the table of context and the scope and obje uh objective of this paper um and and again it's not inclusive of every single other process but you may want to have an appendix item that calls out other business processes that this may call out as well uh most importantly start talking and writing right so uh I I will have given you a very good start on it uh but start talking to other people and in 6 months you actually can have a Ser drafted revised and approved and ready for testing and Road showing uh so hopefully this helps you hopefully it's giving you some good thoughts this isn't

one of those talks where it's like super technical I'm showing you how to do something major and amazing this is a journey I've gone through in a past couple roles I've been in and so kind of just sharing some of my lessons learned and uh trying to keep it simpler and and easier to start and don't let perfect get in the way of good regardless there's always a better way to write a serp and you'll find those make that version two make that version three you got to start with version one though well technically zero but you'll you got to start with version one right um and that's where we where we find ourselves here at the end of uh the end of this

presentation uh that QR code does lead to my gitlab you will see both a bsides Fort Wayne as well as bsides Las Vegas folder uh the LV folder shows this exact slide deck you'll know because I have a colorful uh kimone here uh versus uh more of a a black and white one for the other one uh and it also had a couple uh of those example uh screen caps in there as well um and then it'll also have the um um I think a Word document or a PDF of the act or no a Word document of the actual uh uh example Ser um so and it's got I think company and bold letter

words so you can find a replace and put your own company name in it and you're probably a good way uh on the way to writing your serp already but do give it a thorough read over change things change language have at it but it'll give you a good framework to start um I had to I had to really look and scrip scrimp and scrape together to find to to get a good template going for myself so hopefully that's a shortcut for you uh any questions oh we got a mic a gentleman second row go first of all thank you for a wonderful talk and two things what you have on is that for um

image recognition countering actually very good this probably is a fairly decent image recognition uh um uh uh thing that could probably fool some things uh but no it is not it's just some face jewelry uh this is since we're spooky themed Halloween themed here uh at uh at bsides Vegas I was like oh let's be a Raven Lord and let's uh just have some fun and let's explore uh some ults and other types of uh themes here okay second uh you mentioned signal yes um I assume that you're familiar with disappearing messages on Signal yes I don't know how effective they would be uh with regards to subpoena so can you address that spoliation of evidence is another thing

that uh you know you're going to want to talk to your lawyer about because if you destroy community Communications that could be part of a legal investigation at some point that could be considered spoliation of evidence some lawyers have told me before that if you spoil evidence the jury or judge can use that to say uh there's there's an adverse judgment or Prejudice now against you with regards to what they're so if the other side is saying these things are what those said it's probably they're going to assume that's probably more in their ballpark rather than yours so again talk to your lawyer about using disappearing messages in a chat uh if you're doing incident response out of

band that's a good point thank

you um you talk about tabletop exercises yes um so I'm familiar with uh cisa has a bunch but do you know of any other resources um I can be kind of be like a tabletop dungeon master yeah yeah so uh you also have the card game back doors and breaches from Black Hill security actually doesn't do too bad of a job and you can play it online you don't need to buy a deck you can go online and play that not a plug for Black Hills I just really like it and you can just almost like tarot cards you can deal yourself out uh a scenario and then you go hm how would we handle a scenario like that how

do we envision that would happen here um you can also go to bad things daily uh on uh formally the artist formally known as Twitter um bad things daily has a whole bunch of injects or so this just happened and it's literally just to get you thinking uh tabletop scenarios um but the cisa packages are actually really good and they cover various sectors I was looking at three of them just this past week uh and supply chain and they they do a lot of different things so again another great way to take that and just modify it how you need it to to help your folks um couple of slides back there's a one to five list of things to do how bad

and testings at the end how bad an ID directon would be to do what the software engineers do and write the tests first um and use that as like a understanding how far or how well the rest of it going so you said write the test first yeah so like a test driven development approach has some you know nice benefits would that sort of thing work here or maybe so so if I understand correctly maybe you write a tabletop or see a tabletop first and then you write the serp kind of as you go through that tabl toop is that yeah that could be one way to do it yeah I think there's value in doing that again there's even a

possibility of you've drafted a Ser take the serp that I have here and just go through one of those maybe cesa cesa documents or another tabletop you've seen before and say okay can I answer these questions via this Ser do I know what to do in this case via this serp that's actually a fantastic way to just kind of do those little comparisons and see uh where some improvements need to be made without getting everybody together um again even going to bad things daily and looking through and saying like hm would it cover that would it cover that would it cover that those are very good points uh there so yeah definitely iterative testing is probably

uh open here uh hi um so how would you adjust this for like uh we're a managed stock and we have many customers they have different legal stuff yeah so uh so do I have to make one for each one of them or fun yeah so it depends on the customer I'm sure uh and I've not worked in a manag service sock so uh very much not uh experienced in that um but I would definitely say you probably want one that you have generally available you want to probably work with each customer and say here is our general cyber incident response plan uh do you have one you prefer we use you may have to make some account decisions

as to how much you really want to get into their cyber incident response plan and understand the specifics of it but there's probably nuggets in there about what is a breach for us what is a compromise you probably per account want to understand to know those already so you're probably already starting to customize your cyber inser response but it really just depends is it cyber inser response from the fact that the MSP is getting breached or uh have an incident or is it cyber instant response from your your customer who is using your service just got breached and now you're needing to respond to their incident with them and so those may be two very

different cyber inent response plans right so yeah so just to start going to be a nice template yeah General template and then we'll just fill it in I think that could be a good start yeah all right thank you yeah all right uh one more question

okay um I have a question about people because process technology is easy like we Define our process you know we can buy technology but people is the most difficult part so my question is from your experience how do you get people excited about instant response um legal Engineers PR C stuff how you get them involved and how make them excited about incident response if if people want to be excited about their company existing the next day they'll get excited about incident response it doesn't need to be all scare the children though right um I think uh you know the people aspect again um they see enough news they already know how important this is I

don't I haven't really seen too many companies out there in my personal experience that you put hey we need to practice cyber inate resp response they that they're not going to say oh yeah we we don't need to do that um I don't think there's going to be too many people who need to do that if so uh maybe arrange a purple team uh in engagement where you stage a breach and then start getting them involved you see the chaos and Su and go oh by the way this was a drill as part of our our purple team engagement right and then they might get the uh hint kind of like pentesting right was at one point just

to scare the board into understanding oh yeah take security seriously um you know again the people side of it get them excited about incident response it does take a particular person to engage an incident response actually I don't want people to be excited during an incident I want them cool as cucumbers sometimes you have a nothing Burger an executive is at an eight and you're like I need you at a four but the temperature is really at a two but I need you back down at a four at least because that's where I can handle you you need as an incident leader to be the coolest person in the room right now we're working the problem

we're not working people we're not working blame gaming or anything like that so actually you want people to not be excited uh during an incident cool minds and heads and all that okay all right thank you job everybody

[Music] o [Music]

[Applause] [Music] hey he hey hey he

[Applause] [Music] [Applause] [Music]

[Music] n [Music]

he

[Music]

[Music]

[Music] TR [Music] hey hey [Applause] [Music] hey hey hey hey hey [Music]

[Music]

he [Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music] oh [Music]

[Music]

he

[Music] oh oh [Music] oh [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just try to something this okay you I'm just trying to give you [Music] something I'm just tring to give you something I I'm just tring to give you something [Music] w

[Music]

[Music] [Music] I'm just TR to something okay I do I'm just TR to [Music] something I'm [Music] just I'm just trying to give you something [Music] he [Music] w

[Music]

[Music]

[Music] [Music]

n [Music]

[Music] n [Music] [Applause]

[Music]

[Music] [Music]

[Applause]

oh

[Music]

[Music]

[Music] n

[Music] the

[Music]

[Music] [Music] [Music] [Applause] [Music] oh [Music]

[Music]

[Music]

[Music] [Music] [Music] a [Music] [Applause] [Music]

[Music]

[Music] a [Music]

[Music]

[Applause] [Music] hey hey hey [Music] a [Applause] [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey [Applause] [Music]

he [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music] [Applause] [Music] he he [Music]

[Music]

he

[Music]

[Music] h [Music]

[Music] now [Music] [Applause] [Music] [Applause] [Music] oh

[Music] I'm just something I I'm just [Music] something I'm just trying to something I do I'm just try to give you something [Music] n [Music] w

[Music]

[Music] [Music] I'm just I'm just TR to give [Music] something I'm just TR to something you I'm just trying to give you something oh [Music] w

[Music] a

[Music]

[Music] [Music]

[Music]

he

[Music]

[Music] [Applause]

oh [Music]

[Music]

[Applause]

[Music]

he

[Music]

[Music]

[Music]

oh

[Music]

[Music]

[Music] [Applause] w [Music] [Applause] [Music] just something I'm just dring in [Music] something I'm just TR I'm just tring to give you something [Music] w

[Music]

[Music] [Music] I'm just to I'm just TR to give you [Music] something I'm just tring something sming I do I'm just trying to give you something [Music] m [Music]

[Music]

[Music]

[Music] oh [Music] [Music]

[Music] he

[Music]

[Music] [Applause]

oh [Music]

[Music] [Music]

[Applause]

he

[Music]

[Music]

[Music] e [Music] h

[Music]

[Music]

[Music] [Music]

a [Music] [Applause] [Music]

[Music]

w [Music]

[Music]

[Music] a [Music] [Music]

[Music] [Applause] [Music]

[Music]

[Music]

[Music] w [Music]

[Applause] [Music] hey hey hey hey hey hey [Music] [Applause] [Music] [Applause] [Music]

he [Music]

he

[Music]

[Music]

[Music] track [Music] hey hey hey [Applause] [Music]

hey hey hey hey hey [Music]

[Music]

he [Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Applause] [Music]

[Music] [Music] [Music]

[Music]

[Music] [Applause] [Music] oh [Music] he [Music]

[Music]

oh

[Music] w yeah [Music] o [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music] I'm just trying to get something okay I do don't I'm just try to give you [Music] something I'm just tring to give you something do I'm just TR to you something [Music] w

[Music]

[Music] [Music] I'm just to give you something I'm just TR to give you [Music] something I'm just try to give you something [Music] I I'm just trying to give you something [Music] w [Music] w

[Music]

[Music] everyone we're here with arv and George and they're going to present bug fitti automate your red team infra on the cheap uh just a couple announcements we'd like to thank our sponsors for making bides possible here in Vegas uh gold sponsors uh Toyota um we have uh Adobe and a few others here uh blue blue cat Sim grimp and uh Plex track so uh without further Ado I'll hand it over to you too cool thanks Katie uh hey everyone uh Welcome to our talk uh is everyone having a good bsides this year hell yeah awesome glad to hear it uh so it's actually George's and mine uh it's our first talk so uh yeah it's a special moment and yeah

thanks for being here it's going to be fun so uh with that out of the way just quick intros uh my name's arv and like I said this is George um I've spent about 5 years in offc uh four years in Consulting and then I switched over to the red team here at Costco uh which is where I currently work uh yeah like uh my my nerd credentials like uh I just like any type of World building in science fiction media big soccer for good World building so if you know any nice fonts and stuff I'm I'm all about it so um and most importantly for this talk I am not a Dev Psych Ops person I'm

not a devops person so grain of salt with uh whatever I talk about today so George yeah don't believe a word we say um basically U my name is George uh I've been in offsec for about three years uh that's not to say I haven't had a career before that it just took me about 15 years to get into information security so I was a a convert from audit risk and compliance um and it was a journey so uh I'm really happy to be here it's definitely I think the highlight of my career uh working in researching doing things like this uh and being able to actually wake up in the morning and and really enjoy going to work so um I've

had various it roles been administrator Dev uh risk obviously looking at process looking at all kinds of other aspects of the business um and something about me personally I love I love cooking I I can't say I'm a very good cook but I definitely if I like you I'll probably bring over something that you think is tasty or at least make the best attempt to do so um and then uh again caveat uh I'm not a devack Ops person uh this uh there's there's definitely a lot of stuff that has been uh Googled and researched and uh thank you chat GPT uh definitely helped facilitate this in like a two-month time frame versus a year or so um so yeah yeah big shout out

to open AI um so um I hear uh hackers like Jeopardy so I have a fun little trivia question for you guys next and it's related to the name uh can like we work at Costco so that's your hint uh does anyone want to take a stab at why it's called Buck fitti what is the hot dog yeah exactly it's the hot dog and drink combo a staple of American culture so uh yeah that's kind of where the name comes from and uh yeah like our North Star for developing this was really like one of the quotes by uh one of our former CEOs which was you know if you raise the price of the hot dogs I'll kill

you but and and I hope is there anyone who works at Azure here anyone who works at Microsoft anyone no okay well this this quote is for you guys like you know please don't raise the price of the VMS like this whole talk depends on it so um so yeah let's uh get into sort of the overview of uh buckid you like why did we make this tool right and uh just to like sort of uh set the stage of B uh what do we do as a red team here at Costco right like we have essentially three primary functions right uh the first is red team engagements like spoiler alert and the second is

penetration tests and the output of both of those types of engagements gets you know fed into a reporting Pipeline and we deliver reports to the teams that we work with right so that's kind of like our our main like our main goals uh as a red team so when I first joined Costco and I think GE George has only been there about a year or so more than me uh so I started working there about like 9 months ago uh one of the big challenges that we had with our current infrastructure setup was that it was static it was inflexible and it was opaque right so when I say it was static I mean that the infrastructure stack

itself didn't lend itself uh a lot to experimenting to iterating to building like new components and adding them in uh inflexible in the sense that we we couldn't just turn it off between engagements like it would it would always stay up and uh you know why would we want to waste money on infrastructure that's constantly running if we're not using it at the time right and opaque uh so people come and people go uh you know they switch jobs so uh the people who developed the infrastructure ended up leaving uh Costco so uh what we were left with was this sort of really brittle tool that we couldn't really experiment with but we were super scared

if it ever went down because you know you don't want to break something and not know how to bring it back up without the right documentation so those were the three problem like sets that we identified and that we sought to address with developing Buck fit and and I think these I think I don't think we're alone in in this kind of issue across the industry right like I think a lot of teams find it difficult especially in larger more mature organizations to to be kind of uh have a startup mentality and be able to spin up things uh and have a lot of flexibility and not be part of an oversight Board review something just to have like you know a

VM spun up with a a public interface so um definitely applies it's not just Costco I just want to say that oh and we don't we're not representing Costco like on this top we just work at cost just for the lawyers uh yeah um this is our stuff yeah so yeah we definitely felt a lot like this uh like every time we looked at our infra we were just like how does any of this stuff work uh so uh we had sort of like the choice between reform or revolution right uh we could either like paper it over with Band-Aid solution like incrementally change things around try and figure out what was happening or just reapo the problem

and come up with our own uh you know sort of solution that hopefully wouldn't take up too many Dev Cycles right which is what I'm going to get into next so our goals were uh for it to be cloud-based uh easy to understand uh modular and you know have it is technical debt but we wanted it to be worthwhile technical debt uh so drilling into those uh cloudbase really works for us because uh it allows us to be very ephemeral with our stack we can spin up and spin down as we see fit and you know uh one of the big advantages of using cloud-based compute is that you only pay for the compute that you actually use

right like as opposed to on-prem Solutions uh and uh easy to onboard team members onto uh that's so we made this tool very e very turn key so any like you know even if George and I were vaporized tomorrow uh like other team members on our uh like other team members would essentially be able to download the uh the code and just hit terraform apply and like it spins up right no Arcane knowledge required uh modular uh so that we could experiment like we can plug in plug out different components uh you know just experiment with different tools uh anything that we wanted to uh try out like see if it works for us and then like take out if

we don't like it or keep it if we do and uh we initially we're not a big team right like we're like around eight people so technical debt really weighs heavily on us so we we didn't want to take on technical debt uh that wasn't worthwhile right so we're not trying to avoid technical debt entirely but we do want it to be like a tool that's actually needed that's not just like a hobby project uh you know uh so it has to be worthwhile like it has to be worth our effort and our time so getting into the design philosophy uh are there any devops people in the room Dev SEC Ops okay cool uh so this is for you uh this is like I

am like you know I all of my knowledge about terraform which I'm going to get into all of my knowledge about all of the stuff uh definitely like I got it within a two-month time period so I'm sure we did we we made some mistakes as we made this yeah we need help yeah like so come talk to us afterwards if you know or you know shout it out in the question section uh if you have any doubts or if you see any glaring errors um cool so uh getting into the stack of things right like we're we're this is an Azure Tool uh this and the reason it's Azure is just because like you know our

company's going to pay for it uh that's the that's the only reason uh we definitely like I I'm way more familiar with AWS uh than I am with Azure but uh you know it's this is the cost sync that it goes into so uh that's the reason why we went with Azure and we're we're going to get into this uh at the conclusion where we talk about some of the improvements we want to make but we definitely want this tool to be multicloud like Cloud agnostic we don't want to be locked into a specific uh uh like cloud provider so uh that's definitely in the Horizon uh cheap compute uh yeah we definitely wanted this to be like uh

bank for your buck uh we we wanted to use the bare minimum amount of compute necessary to actually accomplish our goals and this is a really great website um uh I'm not sure if you guys have heard about it but Cloud price.net uh it's kept pretty up to dat and it's sortable and you have pricing for Azure AWS and gcp so it's a great resource it's uh definitely resource we leveraged in terms of trying to come up with what kinds of uh you know virtual machines we wanted to use uh during uh like while we were building this Tool uh terraform was really uh sort of foundational to this whole project uh it allowed us to think mod modularly about

how we wanted to develop uh components off the stack and I thought it was a really neat way of approaching the problem where you don't you're not really stuck with a manual way of doing things you don't have to go into like the Azure CLI and like you know set things up uh in a manual I mean you could script it out but I feel like terraform handles you know the how parts of interacting with Azure uh and we can focus on the what like you know what what is the end state that we want right and that was really valuable to us at the sort of development devel developmental velocity that we were going at and uh yeah it's easy to

iterate upon uh you can just you know experiment like that that's entirely like that's how Buck F was built like we just kept like experimenting with different uh you know terraform config files uh seeing what worked what didn't and yeah it was really it was really useful for us to kind of iterate quickly and I also think it goes to the uh not the tribal knowledge kind of like alleviating that tribal know issue CU you have infrastructure as code you don't have a bunch of guessing like will this run with two CPUs will this need like multiple discs how much RAM um I need to call you know the guy that just left for Fiji quit the company uh to

figure all that out um terraform provides like a kind of a unified kind of source of truth of what it takes from an infrastructure level to actually build up this this platform and the tools that you know this red team needs to continue operations yeah definitely it was it's super readable and anyone can read terraform code it's uh very easy for humans to parse uh so it's definitely useful for you know uh going back to one of our goals which was we wanted it to be easy for our team members to you know decipher uh so it's it was really handy for that as well like anyone can look at terraform code and like figure out what the gist of it

all is um yeah George you want to talk about tail scill yeah so jumping into tail scale I know uh just to give a little bit context take a back step um I know a lot of the the overview of this talk is very tail scale Centric um and when we got the email that our talk was accepted um we were like oh oh crap we have two months let's um let's maybe like do something on top of tail scale so we're not just you know talking about like an ivory Tower this is what we could use tail scale for we actually developed a platform with tail scale being the kind of hinge pinge of the

networking and tunneling aspect of the C2 infastructure and the other like operational related services that we've built into kind of like this demo uh and we're going to demo like what you can do actually live with uh an overlay Network kind of topology uh used for tunneling uh as part of red team operations or pen testing as you compromise assets and pivot within a network and want to like tunnel out or remain silent um that kind of tail scale was the foundation and we're going to build up on top of that in this talk so um so on that note tail scale it's a it's a great service uh it's an overlay network uh we looked at

a few different competitors uh like nebula uh from slack uh tail scale seemed a lot more feature-rich with specific tools and baked into the client for things like net catting uh once you've established a connection uh file sharing was really straightforward within that wire guard tunnel it's all very encrypted uh I I really from the experience I've had with uh using tail scale as a way to access dropboxes within a network it it's it seemed very almost too easy to get in and out of of a an Enterprise environment not saying that Costco was that environment but I would say of the many places I have worked which were fortune 5500 companies I think the the

natur rval aspect of overlay vpns that have coming out recently in the last couple years are really um are really I don't know a lot of ways to avoid that being able to get in and out of almost any network that's routable from the internet that I've seen so um I was excited to start deploying it within um as a as a backbone for other services and I haven't had any problems as of yet yet so um yeah moving on from tail scale uh as a VPN service um we also wanted to kind of stick with the modular context of our platform because if we did all get vaporized me and arv as The Architects um we work with very

competent people and and techn Technology Savvy um folks on our team but they they are amazing at things like develop malware uh they really want to focus on what they love doing on their team and we love doing architecture and solving problems as well as doing pentests and other things but uh we found a niche that that really was a support role with building out tools and and talking to our operators and asking them what what would make their lives easier and then trying to build that out within Azure or whatever and and Docker seems like it was a perfect method of getting an idea from an operator uh finding a like a proof of concept open

source solution to that that someone spun up uh making sure that it's secure enough to implement for operators so we're not piggybacked on by real malicious actors uh and then deploying that quickly without a lot of uh development overhead like on the operating system level of a VM so uh we could pick and choose from a few competitors for a specific service throw that into a platform and have it play nice with other Serv Services uh with very little overhead and that can be easily translatable in the event that somebody does leave the company um to to someone who's technically Savvy knows what the service is doing and can kind of like you know Google the rest of the

way there to either spin it up replace it or you know update it as needed so um Docker really helped with this environment uh and it's also integrates really well with terraform and that's pretty much why we chose it open source components in general um we we wanted to stay cheap we did this research on the side it wasn't sponsored by Costco um so we wanted to stick to the B 50 rule very closely um we got it down to about 150 a month or so if like you're live no not even that 50 bucks like 50 bucks yeah okay wow I'm I'm glad I didn't open up Myer account it's it's all our up um so yeah about 50 bucks a

month to run like a fully functional platform form that you can do pen testing off of or red teaming um with a lot of Open Source tools so like zero cost on the open source tooling and amazing Community out there for security especially like the the even the like tail scale op Source their protocol and within a m like a year or so uh there was just really great tools out there that employed that protocol uh and you can spin up like lookalike copycat tail scale clients and servers uh so you didn't have to go through tail scale.com you could have as many users as you want um if you had the kind of the motivation

to configure it all yourself so you definitely get into the weeds with open source uh the uh the resource Group delete everything and at the end of it uh you're going to see me yeah like right there it took like what like 2 minutes and yeah like uh the whole stack is down again right so so super easy to spin up and spin down uh really flexible uh you know so if you have an engagement get started up at the beginning of the engagement uh with super little latency and then at the end of your engagement you can just tear it down uh as quickly as you spun it up yeah and and definitely definitely definitely

remember to take everything all your reports off the servers before you do like print out the report get into the executives and then tear stuff down because you're not going to get it back yeah it's lost to the ether um cool so so yeah now we can uh show you guys like uh the live demo we already spun a stack up uh so uh George is going to just walk us through like what what it looks like under the hood so I'm just going to have a moment of silence for the demo Gods okay no it hasn't broken yet I just wed it off um so so this this what's going to be on the repo publicly is kind of a sandboxed

version uh I you know the it was configured figured to kind of be self enclosed uh initially we didn't even want to open up to the uh to expose anything to the internet is kind of like a uh an environment that you could play around with in Azure to just get a taste of it and see how you liked it and if you wanted to add stuff to it and there's going to be there's definitely a good amount of configuration you could do to lock it down to get everything self-id search enabl to get um you know let's encrypt or CA scerts um on the uh the domain names that you would buy to to then connect to this platform from

external devices or if you want it to be working remotely and attacking like a network um you you just you just this there's a lot to be done on top of it but what we did uh we just kind of uh bogarted the name Buck 50 on the Azure domain um for West Us 2 uh we uh tweaked this a little bit so that we could access it externally so I'm accessing from a web browser um and the team buuck 50 are all kind of dnsa records that are built into the head scale configuration uh and and you can modify that as you see fit uh but it does take some kind of reversing or at least

looking up and Googling like how to install you know head scale server um what do I need to do if I want my own like personal records for devices and call outs so um once you get it to that point um make it your own a little bit uh you can be presented with like this dashboard so I set up three different dashboards I have a default dashboard for someone who doesn't even have access this is a public view dashboard for a Noob on your team um you totally mess with them uh and uh have them do like a checklist onboarding checklist it's just it's kind of like the gloss of having a dashboard service versus just having a

static web page with some buttons for you know your other tools so um we got the orientation page then we have like an operator page this has the services uh added some bookmarks that are kind of relevant to infos SEC um integrated with slack um this one you can just like open opens up the app Discord uh and then you can access like Ghost Rider G T you know the Pacman Services we talked about um but you can't really administer anything so I set up a third one where you can actually go in and change the reverse proxy you have uh links to the administrative portals for all the services uh and you can really start

customizing like when you add devices to the network are they going to be uh like special case systems or are you going to add like more infrastructure where you want to mess with engine X and and and have Services route directly to that um uh and and uh now we can start getting into kind of the administration part of the thing so head scale um kudos to head scale uh it took it was definitely a little bit of a learning curve for me um I'm sure not for a lot of people but um getting it set up so that it was as tailscale decom um like intuitive and usable uh it it took a minute um but we got there uh

we got a web UI uh front end install on the on the heads scale command kind of CLI interface it's an additional service you don't really need it if you're comfortable with CLI you can just SSH into the management server and and start running uh scale CLI commands unlike Docker exe um in the container and add devices register things create preo Keys um some of the some of the some of the things I like about this web UI is just like like graphically like you can if you had a bunch of um if you were trying to like just give a link or or some screenshots to to an operator so that they knew okay you your devices

are on this subnet you can route to like the 10.6 and 192 on on this part of the network and then like 172 is going to be another device on another part of the network you can really quickly reference that with the overview um but essentially um head scale and and tail scale the client acts as its own DNS server so um the gooey client like uh I was kind of like is this too much is this going to be too much overhead is too much fluff for engineers but um this this this client really kind of streamlines a lot of the the command line commands that you would want to use as an operator in my opinion

like uh accessing exit nodes so exit nodes is a feature of tail scale where uh since since it's a a distributed VPN architecture there's no centralized VPN server that you're connecting to every client can itself be a VPN server and every other client you can have multiple exit nodes enabled uh and any other client can then use any device that they wish at any time independently of the rest of your team um to exit node your traffic through that device so uh basically spontan you can like a you can have ephemeral VPN and servers as soon as you get a uh a Dropbox or a Target compromise and you set up the client on that system that

system can then start accepting traffic from the rest of the team uh and Route it through that that Network and uh effectively start pivoting and doing it whatever you want and you can dynamically shut that down at any second um and uh yeah it's just it's it's just very um very convenient and it's all over wire guard so it can run silently like you really like there's no there's not going to be any logging of connection attempts to the VPN server like in a traditional VPN once once everything's kind of inhouse on the tailet network connections are only going to be like connection attempts will only going to be visible to the devices that you're either connecting to

or from so um even other devices other operator devices they don't even have visibility into what exit note I'm going to be using and like why I'm using it what time of the day I'm using it uh and one of the main motivators for doing the op Source head scale build despite the learning curve is that every connection attempt like if you're if we're going to use the Enterprise version of tail scale and pay the money and have as many users as we wanted um they're we're still using their coordination server uh which kind of allows the whole natur versal aspect and makes it really easy to get in and out of networks um but tail scale

also is going to have records of those connections so what we did is we bit the bullet and and just kind of built our own coordination server uh in in-house Tales scale.com completely uh so it's not just like a money thing it was like if you want to be silent and you want nobody to know and you don't want to be part of any derp Network um that other people may compromise in the future just because there's not a proof of concept now doesn't mean like eventually there's going to be a big breach um that was a huge motivator for us to get this totally like insourced within Azure and uh basically run completely silent it's

like red October basically Bally um so I think I plugged head scale enough and tail scale um moving on to Ghost Rider again a super important like unsung hero of the whole pent test process uh report writing uh you have access to this uh this environment uh it it you know it looks kind of un like the first time I open this up I'm like oh 1998 I'm back like hi um but uh like looking into it like the integration with SSO like groups you can create through keycloak to really pin down um all the access to the different findings the different reports like it it uh I have some familiar with Plex track it it it gave

me everything that I could imagine from like most pay for subscription report writing services uh so uh really happy with that again easy to get to from the console like all these are just like those DNS records oh yeah and that that's a really handy thing about uh our head scale implementation or a head scale implementation because it has its own internal DNS functionality you can you can make these really easy to remember internal sort of URLs so all of our URLs are just like you know team. bufet for the dashboard or get. buf or P.B FY so it made like it makes the usability of the stack a lot better uh for the average operator on our

team yeah one of one of the reasons I I think we might not have our our code cot up yet is cuz I'm going through all the comments that I wrote as I was troubleshooting like configuring stuff and like trying to like you know Easter eggs too many Easter eggs too many funny jokes uh just need to cut this out and sanitize it um but long story short uh SSO keycloak uh if you want to do something like authentic and really cut down like just your bare minimum you want authentication services and SSO to these Services which I highly recommend you do uh especially in a group enironment it just makes collaboration much easier uh and spinning up of of uh

user accounts on those Services kind of seamless um with keycloak it was a lot to digest at first but uh um eventually we got there we uh there's a lot of um capabilities for MFA especially um using devices RSA tokens um but and also setting up user profiles so I like being able to have custom attributes where have SSH keys can use for later um but yeah SSO and a nutshell very important keycloak um highly recommend but good luck finding documentation out there it's it's a little bit tough uh and then lastly uh not lastly but we got a couple more services um the GitHub the gitlab uh gy uh Service uh definitely pretty important uh adds Wiki adds more

collaboration services for your team it's also a secure environment to store code uh without having to get approvals or um you know from from the board of directors or whatever to to have malware in your environment so um pce spin kind of self-explanatory and that's pretty much it

cool oh we do have one last slide to get through uh so just in terms of what's next what's on the horizon uh definitely want to make this multi Cloud uh don't want to get locked into Azure at all uh uh we also wanted to one well and that dovetails very nicely with what we also want to do which is kind of like a python wrapper around the whole thing so it's kind of like a like a wizard experience almost that you can just like go through and uh you know it like it'll dynamically alter the terraform variables for you so you don't have to go in and manually fuss around with them um and one big one is kind of

like parallel stack deployments right now we're limited to one per subscription we definitely want to move to a uh to a place where we can have multiple Stacks running at the same time uh uh because you know like our team is pretty small we we don't really do multiple red team engagements at a time but I'm sure other uh teams do do multiple pentests or engagements um uh in parallel so we definitely want to implement like a a parallel sort of uh deployment model uh also want to like bake in like known good uh images and then just deploy them on the VMS right now we run a lot of inline commands on the VMS upon startup and we want to move

away from that just clean up the terraform code base a bit more so that's like so that you know you can abstract all of that stuff away into the actual image layer of the VM and then you know just use a registry or something to deploy it um also for the pentest VMS like the Cali Linux VMS we want to Implement some sort of VNC client um on their end so that you can just use your browser to access them uh yeah sort of like if if you guys have uh played around with hack the box or offsec uh the way they have like uh browser based uh VNC Solutions we want to have that

implementation as well and uh this one is real like pie in the sky sort of thing just like a local sort of like uncensored llm uh uh model or containerized model that's running uh uh on maybe a GPU cluster or something that'll help you craft uh exploit code and stuff uh without having to resort to you know chat GPT or you know Sonet like which have a lot of guardrails around trying to generate malicious code so we definitely want to explore that but that's definitely like uh you know version 3.0 or something and it's definitely going to be like an optional uh sort of module uh in in the future stack and that's about it uh so thank

you very much uh like like I said it's our first talk so we're so glad we had this chance to talk to you guys uh so uh yeah our our email and Twitter is on there so uh feel free to hit us up anytime with any questions or comments about any of the stuff and like I said the the git repo is also there and uh if you want to bookmark it and like you know revisit it later in the week it should have uh a work a turnkey stack for you so you can just download it and get going with it yeah we we promise we promise uh yeah I don't know if we have time for

questions but yeah maybe one or two okay one or two questions uh yeah how many have you run this infastructure did you hear the question you sorry could you repeat that how many engagements have you run this infrastructure with how many engagements have we run this infrastructure with as of now zero uh we just like sort of operationalized it maybe a week or so ago so we're looking