BSides PDX 2023 - Come Together: A framework for a shared security language (Lea Snyder)

[Music] okay so who am I um I'm a principal security engineer I work for a you know Cloud company like a lot of people do um if you recognize my name it's probably because I actually do a lot of community work um I'm actually one of the bside Seattle organizers so I love bsides it's an amazing amazing uh organization um I got two links up here if you want to learn more about me the first one is actually my tldd SEC uh staff plus security engineering interview that's a mouthful um if there's anyone else in the audience that's like a staff plus engineer and wants to be interviewed feel free to come up and talk to me it's a pretty cool experience um okay so this is my question we're all security people we're all interested in security but do you ever have that like nagging feeling that you guys like aren't actually talking about the same thing when you're talking about the same thing yeah we've all had that all right so welcome to bides there's audience participation um if you really don't like it just you know keep your mouth shut it's totally fine um okay and we're not going to like run around with a mic just cuz like this is a lightning talk and that would take a lot of time but does somebody want to throw out a definition of a vulnerability like just shout it out oh come on there's got to be somebody exped something that can be exploited is that what you said I got I got one in the back too all right I I don't know how to top that all right so cool we got a definition of vulnerability can someone give me a definition of an attack assault assault is that what you said well I mean that's true but let's let's do it in a security sense exploiting a vulnerability method a method okay do you see how like we just said that an attack is exploiting a vulnerability so then if we weren't on the same page with a vulnerability we just said we were going to yeah all right but here's the example that will probably piss people off a little what's crite scripting okay so is okay it's an injection what was that violation of trust it's a coding error it is a coding error so but here's my question for you is it a vulnerability or is it an attack it's feature okay I love this audience this is great um but like honestly depending on your Viewpoint you're going to say it's an attack because you're a red teamer like how do I get into this thing oh I found some cross scripting yes I get the data or your blue team are being like a for love of Pete why didn't they sanitize their inputs and encode their outputs right like so that's the problem and so we tried to kind of like how are we're going to solve this problem right and for us at the time we're actually facing a slightly different problem we had a boatload of data anybody else have a boatload of data like just like just ridiculous amounts and can you pull it all together so here's the problem let's say you have pent test reports you have threat modeling results you have detections what else is there's red teaming there's maybe a risk analysis you've done you've had an incident you've done a post incident report you've got all this data it's awesome it tells you everything that's wrong right uh how do you pull it all together and the reason I asked that is that so often we each have our different way of defining those things and now if you're me and you're trying to understand the landscape you're going how that I can't can't join like I literally sat there being like I can't there's nothing to join on like how am I going to do any of this analysis and so this is what I talk about it's like a to me it's a building block right so it's like this little tiny thing right here it's so important that we all ignore because it's just easier to do our jobs all right so what's our approach to solving this um and the ground tradition of like we all have busy day jobs and we all don't really want to do data analysis we formed a V team so that's just a volunteer team it was like no one's job we just all volunteered and said we're going to do this we're going to solve this problem um but I just said we all have day jobs right and so then I had to go make the case that we had to prioritize this effort and here was going to be the results and here's the impact and you know you make the great program management case um so we started to build our vulnerability taxonomy that's what we started with so what so what was our goal well selfishly for the engineers it was to accelerate our data efforts right like that's why we wanted it um and then we realized it might be helpful to have a shared language when talking with our servers teams about their vulnerabilities like it's kind of an afterthought like like seriously we're all like oh we can analyze data this is going to be amazing and then we're like oh wait we can speak better with the people we help oh cool um but as usual you have a lot of data and I'm going to analyze it and you're like oh no now I have more data so you know pick your poison so um I wanted to share sort of like the large categories we put together bit of context um we support teams that build identity and network access software so we needed things that were probably may not apply to every team we also got to avoid things that didn't apply to our team right there are definitely uh categories that are you would see that are missing they just don't apply to us we don't have to worry about them right but these are the ones we sat down and did and I want to kind of call your attention to the other um that will be the category you both love and hate it's so much fun all right so let's like I I Hadad of def find one of the categories that didn't have a bajillion like item so I could actually give you guys an example um these are both basically you know denial service let's be real um and so what we triy to do is talk about it in terms that people might search for it right not just us as security Pros but like denial service is something you know other people know so we try to include language that could be accessible um trying to see if there's these aren't like maybe the best examples but we talked about like um when possible we try to include like fixes these are obviously a little less fixy related but that was kind of the goal is also like what's a fix right um but as you see denial service so then what could happen as the service teams like I don't really know which denial service I've hit that's okay they can come and talk to you at least they we're starting on the same page that's exciting right okay so how do you actually so you've done all the work right and by the way to do all the work there were originally like four or five of us who sat down and said okay how do we start how do we build this and at our company we had actually a large vulnerability taxonomy it had 800 individual items and we thought to ourselves there is no way anyone's going through 800 items to find what their problem is like that's just not going to happen and remember I said earlier there's just stuff that doesn't apply to us so we paired that way down and then we said what's missing and then we went out and looked on the web at other people's examples cuz like why reinvent the wheel if you don't have to so there were four of us we did that kind of work together and then to actually write the definitions because like if you look at this we've got a subcategory but we wanted like a description that was tangible right that was about a team of 10 but that like we ended up with about 200 and with a team of 10 that's only 20 definitions of person right you can bang through that um especially like I don't know if anybody else's days like mine where you have like Mee meting me mee mee oh I get 25 minutes okay what am I going to do with 20 I don't know what anyone else does with 25 minutes there's not much you can do but I can't write a definition right like it's it's like this is the perfect work for somebody as busy as me all right so what was our goals when we put it into practice we wanted to make it easy for teams to adopt so for us that meant integrating into all our tooling so like that I have to say if you write something flat congratulations you wrote a flat document don't expect anyone to use it um so ours is integrated into all our tooling there was a lot of work to make that possible but literally if if a team's filing a bug there's just pull down menus right super easy that way then we can pipe that back into our data analysis but culture changes hard um I will say if you decide to go down this path you really need an evangelist you need somebody who's constantly being like hey you didn't fill out the you didn't fill out this category or the subcategory uh can you and then do you need help because if you don't do that there's it's not going to happen I got to be honest and like it takes a while to train other people to do what you want right like they just they don't want to do what you I mean maybe they do that's not been my EXP experience and then remember how we talked about the mysterious other um I didn't put any parameters around it when I introduced it turns out that was a mistake that I learned the hard way um I started getting things back like external researcher and I was like how is an external researcher a vulner like I just sat there being like I'm so oh this came in through bug Bounty uh that's not the vulnerab like I'm sorry an external researcher is not the vulnerability let's be very clear about that um and then sometimes people just put the most like detailed thing I'd ever seen and I was just like but there's a category that says almost this for btim why are you doing this um so we ended up having to do some cleanup we ended up having to write some guidance like please don't put external researcher it's not really a vone um it's important though right cuz there's stuff we're not going to think of and the threat landscape changes the attackers get smarter and we're going to have to like edit it um the problem with that is if you run metrics it gets a little dicey as you change things just calling out there just you know warning you um so how do you build your own because I think that's the important part right like that's what we're all here like how do we do this um Define the problem and then Define the problem for your space right remember how I said at the beginning we had a great list it was 800 items that was not I still think that's a lot like every time I like reflect back on I'm like wow it's a lot of categories um You probably don't actually have that many right like you you you well maybe you do and then good luck um but like do like Define your space and your problem and figuring out what your tooling is what kind of changes you're going to have to make what all tools do people use and ask them because here was the funny thing I learned along the way there are a whole bunch of things that were not on my radar right that people are using and I needed to get those changes to those tools as well um and like 100% borrow from other people like do not go and reinvent the whale like just don't like there's smart people we're all smart people let's be clear but I mean bug crowd and hacker one have their definitions yes they're they're obviously about bug Bounty there'll be things that won't be on that that you'll need but it's a really good starting point if you prefer to come in from the attacker perspective um miter is like I mean to me is like one of the best ones out there um we're actually in the middle developing an attack taxonomy as well because we've learned that that may help people understand the vulnerability better so I do think like having both is actually really critical to success um do not the love of God let the perfect be the enemy of the good um what we discovered as we started editing is that people were doing that and I was like just let it go like you would get on a call to discuss definite because we would review all them you want to make sure they make sense you know if the Call's gone up for 30 minutes and you're still on one one item you're doing it wrong I mean you're not you're having a really interesting debate amongst yourselves and you're learning stuff but again like it's got to just be kind of simple like don't over index on making the thing perfect and be okay with iterating because that's exactly what we did we just kept iterating and iterating we still iterate one of the interesting things is as we started to work on the attack taxonomy we were able to then compare it to the vulnerability taxonomy and say okay what are we missing here and that was actually a really interesting perspective that I recommend um accountability really matters so that's that's that evangelist person constantly poking at people being like hey you didn't fill this out hey this category really matters hey can you do the thing um it sounds really annoying but it's actually kind of fun right because you could just basically like I don't know about anybody else it's like oh I got I got brownie points because I said you didn't do a thing like I did a thing because you didn't do a thing cool um and keep iterating like and be open to that and be open to like people giving you feedback and telling you what's wrong with it don't take it personally we definitely made a lot of changes based on Behavior we observed um if you use internal code names for things for example I think that's actually very common like we always see this type of incident we call it this code name so externally people don't don't know what it is put those into your definitions I learned that one the hard way people kept telling me it was this type of incident and I was like what I don't but there's a bone okay um so if you see like repetitive Behavior especially by like your service teams go fix it this has really helped us actually a lot like we actually can now run metrics and say okay here's the most common category of problems here's the most common subcategory of problems like we actually know what we need to go solve for and what I think is nice is not just incident data right it's also your threat model your risk analysis it's like you combine everything to actually give you that full landscape I think when you don't do that you actually miss things um all told I say the journey took us a month and a half um as I said originally for engineers this was not our full-time job we have a lot of other things to do um you can go really fast you can go really slow there's no like at time I wouldn't think um and that's all I got for you and this is one of my favorite ones out there I mean come on it's funny right all right does anybody have any questions I know that was sort of like a whirlwind tour of our experience okay get The Walking microphone here so how many Engineers are you supporting or how many how many uh how many people are you dealing with having to run down their their vulnerabilities or attacks that they're that they're logging so make sure I understand you're asking how many people I support that are going wow you know I've never counted um we actually let service teams put it in so it's in addition to security team which is about 60 of us we we support 260 some odd services I don't know the math a thousands some odd engineers and product managers like we let anybody actually put the data in we spot check it I realize that means it's not perfect and I'm okay with that so when I was uh watching your presentation what the first thing it jumped into my head and you kind of I think hit toward it towards the end for you know don't recreate the wheel was it sounded like you were ex um describing the cwe and I was curious if you looked at that and used it and how much that played into it and how much you ended up differing from that if that was part of your uh your research and where you started it's interesting we didn't use cwe it's another one that I've thought about after the fact that would be useful but we basically because we had an internal one we could start from that's what we started from and then we pulled in some stuff from I you know I honestly don't remember it was bug Crow or hacker one like it was one of the two big bug bug bounties we went and pulled in some other stuff but CW I mean that's a good point I have that up on the slide that's another great place to start um like that's my point there's all these resources just use them hey long time to see um what I'm curious about is what types of like data visualization or analysis or like do do these reports go up to the leadership or like how how is this like you know kind of weaponized within the organization to make it more effective okay so the best best part is like I should have said at the beginning like how many people actually know me how many people work with me so that's a funny question to me for that reason um we actually don't have a ton of visual visualizations around this it's mostly for the security team I'll be honest like we're we haven't been great about sharing it that's sort of selfish of us because what we care about is like you know as you and I it pains me to use this expression as you should left um you really want to focus on prevention right and you want to focus on protection but you're not going to build those right if you don't understand how people are attacking you so for us it's more like our own little secret store of data we just make all the service teams tell us things but to your point it's probably better if we share it but we have not actually like shared it with the organization I'll be very clear about that you've planted an interesting seed maybe I'll consider it um we have other dashboards that we think are more appropo this one as I said we it's for us it's selfishly for us oh we got a question up here so yeah my question is um I've always been a big proponent of like adoption as king kind of goes in line with that um some of the struggles that I've had with the it career has been um when standards are set they're not enforced so they become kind of useless and that other category becomes like 99% like other but also the description either has very little or it's a word salad so as someone who's not in leadership what paths have you seen that are effective and helping the Evangelist or the Champions push forward that yeah so as I said like you definitely want people who like I'll be honest I took on that evangelist role pretty hard like pretty hardcore I was like I'm going to make everybody fill this out gosh darn it I work so hard on it and it's so important um but I can't be in every meeting right so I can't I can't always make sure it's happened so then it's about the entire security team like coming together and making sure this happens so if we get things where it's missing we need to send it back um we really just need to encourage people it's okay to be wrong or if you don't know come talk to us like we're actually friendly um for other like yeah other was difficult as I said we did not set any parameters around it originally and as I said I like my favorite one was like I mean and if it was just one person telling me an external researcher was a vulnerability I would have just chuckled but when it happened like 10 times I was like oh no I need to I need to fix this um I would definitely push back on other so other is the one that we do tend to like when someone puts it in we tend to push back pretty hard on it because to your point they either don't tell you enough information you get word salad or like it becomes everything and we're really like please make sure if you're going to choose other that really these 200 some other ones don't apply to you and if you don't think they apply don't apply to you come talk to us before you put it in so it's like not the best mechanism because it does require people to like engage but you actually get really fruitful conversations out of thos