What Does it Mean to Build a Proactive Security Culture in an Organization

hey everyone uh thanks for joining our talk uh we're here to talk about what does it mean to build a proactive security culture um in an organization um well my name is makund I run product security at chime uh prior time I was at Credit Karma and even proud of that other synopsis or Curry hi everyone my name is arcadi I'm a security architect at chime and previously I was at Twitter and Airbnb cool um so what are we planning on covering today very briefly we're going to talk about what do we mean by security culture and why invested it at all why is it important then we're going to talk about what has and hasn't worked for our

organization and our past organizations then a little bit on how you can measure security culture and then our concluding thoughts and questions from you all right so what do we even mean by security culture so I think at its core the culture that you have determines how work gets done in your organization so this is very much rooted in one's Behavior so uh things like how people work how do people communicate what behaviors are incentivized all of these are some or some of these are factors that contribute to what kind of security culture one could have in an organization and we thank all of these are factors at one could consciously uh choose to invest in we'll talk more

about that um so why should you invest in culture why is it important well almost all security work at organizations happens outside of the security team it's going to be done by product engineering data marketing analytics et cetera all of these other teams they're the ones who are really in the day-to-day and implementing all this work and you're not in their reporting chains you can't order them to do things you can't tell them to you must do this thing so it's really important that you lead in your organization through soft power and through influence and you do that by having a security culture that's one of the most scalable ways to enact security changes in your organization and the

business needs to find Value in your services so you should make a conscious effort to build social capital with people and you can use that as currency to get your changes accomplished um and it's not always about getting things done you just need to be reasonable sometimes um so uh this is a tweet that really resonated with us hey if you work for a security organization in a company and your user see you as the enemy maybe you should rethink what you're doing you know if if people don't likely don't like you then you're gonna have a very hard time accomplishing your goals cool so other reasons why you would be investing in security culture I think um

you need your organization's help to get done through your regular day-to-day security work most of the time security teams are tiny compared to the size of engineering or the rest of the company so if you look at engineering it's close to like one is a 150 uh the ratio that is like for every 150 Engineers you have maybe one security engineer so there's absolutely no way you're going to know what all of these 150 Engineers are doing uh you got it you've got to build a culture where they trust enough they trust you enough so that you can they can come to you and tell you when for example they're going and writing a whole new third-party integration or if

they actually accidentally clicked on something or they don't download something that they don't actually know if it is safe and they they should trust you to come and tell you that hey um I did this accidentally um similarly like having a collaborative culture here makes makes it very easy and not an upward battle for every single thing you don't have to justify yourself just because you hey why are you doing this if you are caught up in that world then you are never going to be successful um growing that security program cool so what works uh building relationships and both top down and bottom up I think both of them are equally important I'll go over top

down um so if you have been at situations where uh you've had a hard time getting the rest of the company being accountable for security uh that means something is wrong so I think start with the leaders talk to your VPS talk to your senior directors talk to your directors ask them what is it that they are worried about security like you've got to make sure that the question is turned towards themselves and ask them like hey reflect on what is it that you're building and what is it that could go wrong from a security perspective we are here to help you but we need you to tell us what is that you're worried about

um similarly as what are the top goals uh try and convince them like how are you there to help them uh satisfy those um also understand how are they incentivized by if uh supporting security or getting security stuff done is not part of their incentives and you're never going to be successful um this is for more advanced companies or sorry teams if you can go do security Road shows so um I found this really helpful um at the minimum you'll figure out who the most toxic leaders are in the rest organization um why is that important because you'll know where you're going to get most of the resistance from um cool so that was uh building

relationships top down um I also find building relationships bottom up very effective meet Engineers where they are so be very active on code reviews on design reviews in jira on slack in your security ass Channel um everywhere else I find it really effective to be very vocal very active and if people can see that you're putting in all this work and being very responsive and watching things very closely they'll understand that it's important they'll be more likely to work with you they'll be more likely to listen to you if you approach them with some sort of change or ask request or something like that you know sometimes um certain teams you might have a pretty

good relationship with the leader of the team and you can go to them directly but other times you might have a poor relationship with the manager but you might have an excellent relationship with a an engineer on the team taking that bottom-up approach and you can just go to them directly and say hey can you patch this thing really quick and Skip all the rigmarole of like a Sprint planning or something like that so it's really nice to have that kind of optionality and also for more advanced teams uh if your team company reaches a certain size having an embedded security program where you're actually going to take a security engineer and put them within some other team also can be very

effective again requires a certain level of maturity to get to that point cool uh what else works so having a challenge Network definitely helps so this is a group of people that you trust that can give you a good candid direct feedback um realize that they are going to be your biggest critics but it's fine like in have them and whenever you're building a new security product or a security feature or process and ask them for feedback like tell them hey how do you how how does this look for you like does this make sense um I think they are people that you can rely on and I think having that challenge Network in every company of

where I've been in has really helped uh me uh growing that security culture similarly I think uh one other thing that a lot of um teams get on into the world is trying to make everyone happy uh just realize that's never going to happen and um actually it kind of degrades your security culture if you get to that level um so I'm so it's perfectly fine to make sure that to realize that not everyone is going to be happy with what you do um another tool that has worked very effectively for us is just being completely transparent with engineering with the rest of the company share metrics share bug Bounty reports sure pen test reports of oh is it not Audible

okay I'm just going to use this for now um so uh yeah share metric sure bug Bounty report sure pen test reports um I've actually worked in security teams before where maybe uh they're worried about like having their vulnerabilities be open in jira maybe somebody sees the vulnerability and then they exploit it somehow I'm not saying that's not uh an impossible risk but I think that the benefits of sharing and being open with your program far outweigh any sort of risk from someone like exploiting an issue that they find in jira so leave all your issues open run blameless Retros um admit when you're wrong show vulnerability not just the security kind you know sometimes you misjudge the

impact of vulnerability if that happens just say to the team my bad don't try and like uh force the issue and then later if you're approaching them about something else they'll know you're coming from a place of good faith um share updates and be visible so that means presenting at company all hands and giving talks sending out a newsletter sending out presentations um doing engineering Brown Bags very common companies um one thing that I find really effective is bug Bounty walkthroughs I think a lot of Engineers actually really are very curious about security how does it work how do I get into it but they don't know or they don't have the resources available to

them so if you just send out a blurb here's a vulnerability that we had here's what the impact was here's how you exploit it here's like a curl command that you can run Engineers will really enjoy that kind of content and then they'll be much more willing to work with you and again more of an advanced item but if your company reaches a level where you can run security awareness events um that's also very effective it requires a lot of work cool I think making it easy to do the right thing is also really really effective a lot of companies are uh using the Netflix paid path approach um I think it's proven to be effective

um like make sure that security controls are baked into the uh Frameworks or libraries that your rest of the engineering is using and when that's not possible make it easy for uh people to be notified that hey something is going wrong like here's how you fix it simply um similarly I think documentation and education that do not suck or as directly to the point is really really effective um and I think instead of Toronto teach them the entire history of a vulnerability and everything just get to the point and like make it very effective and that really works um and gamification uh is also huge uh at time where we are uh if you have

missed the previous talk on Monaco it's basically a tool that we've built inside that basically gamify security for the rest of uh services that are all the rest of the things that are Building Services so we have say hundreds of services uh we want all these services to do X Y and Z instead of going and reaching out to all of these services to say hey go to XYZ we gamified by give them a security score uh and that's been really effective um the other thing that has really uh worked for us is meeting developers where they are like if you onboard a dozen vendors or you are integrating with a dozen tools you can't expect a

developers to go log into all of those tools and look at all the findings like uh simplified for them meet them where they are use graph that's what the developers are using use GitHub Advanced secure dashboard if that's what people are doing like make it available to them where they are instead of expecting them to log into all your security tools um similarly I think how many of you here have a policy that says uh you shall or you must rotate your secrets every 90 days right quite a few of you have that I don't know if you make it easy for developers to do that like how do you hormone help themselves service tools

that they can go do with themselves right so almost no one so I think that's kind of the point of Tomorrow makers like make it easier for developers to do the right thing and they will more likely go into that right thing um another uh way that we find it effective to build security culture within an organization be pragmatic um don't practice security absolutism be realistic about risk you're never going to solve all risk there's always going to be some residual risk remaining don't let perfect be the enemy of good you know you can resolve some risks really really effectively uh don't uh practice security absolutism um don't be a security Nazi if somebody comes to you and asks you a question

maybe even if the answer is no like sometimes you have to say no but don't be mean about it be empathetic trying to understand their needs and help them accomplish their goals um have a security engineering onboarding or just regular security onboarding for your whole company that doesn't suck and be relevant so you know I've seen security presentations in the past or and I'll touch on this in some later slides but they'll say things like don't click on suspicious links which is like I've been doing this for a while I don't know what a suspicious link is um don't uh don't connect to untrusted networks like I don't think that's really relevant anymore um make a don't don't practice like

performative exercises right make it relevant for people show people some real world impacts show people some real world like how are companies getting hacked um and then and then you know people will listen to you um they'll be more likely to take your advice or work with you and this is just a funny comic that I enjoyed from poorly giant poorly Drawn Lines um you have to choose your battles I choose all of them like be like the rat don't be like the The Raven cool um also yeah you rely more more and more on your other engineering or other domains of your company to do the work like give them credit give them Kudos

give them shout outs send them cupcakes it's perfectly fine people love cookies um I'll take more if you have some uh but I think uh in general I think uh people tend to really focus more on like telling their team like hey you're doing a good job for going and doing all these things I think it's also important to tell uh your vendors or your um your other supporting functions in the company that hey when they're actually doing things like tell them thank you um also realize that not everyone is incentivized the same way so see what it is that the teams really and values like someone someone just value more bonus or someone might just be happy with the

cupcake it doesn't matter like just find out what they are and uh reach out to them with us so only I think it's also okay to ask for help a lot of security teams get into this world of oh and the engineer I need to go solve all the problems like no I don't think you know the domain as well as your domain teams too so reach out to them tell them what the problem you're trying to solve for um how is it you're gonna um how is it they can help you solving that particular problem and um actually it actually works uh wonderful uh cool so that was some things that have worked well for us now we're going

to go over some things that have worked not well that have made the security culture worse in our organizations um so to start things off I'm just going to give you a second to read this tweet that I enjoyed um I think it's a really funny tweet but the reason that I included it is because I again have seen this in so many organizations where people don't trust their own employees they don't trust Engineers they keep everything very close to the chest and you know you can imagine if you go to some engineer and say hey I need you to do this thing it's really important I need you to drop everything and do it right now I'm not

going to tell you why I'm asking you to do it like that's never going to land well but I've seen people do that it's nuts so it's kind of just the opposite of the transparency slide you really really want to be overly transparent with your team hard deadlines and ultimatum Zone work don't try this um there's always going to be scenarios where your policy doesn't work or your policy is not applicable so try and reason with the engineering team or whichever domain team you're working with um for more more more often than not we have seen that people will try and help you do the right thing it's just that if someone is going out of the way to not

do something it's maybe it's because whatever you're asking them to do is not applicable for their domain um so yeah um another thing that doesn't work too well is being disconnected from reality or losing credibility with engineering so examples of that if you're going to run some sort of off-the-shelf scanner and just like open up hundreds of jira tickets against the team and ask them to validate the findings that's going to be really hard right a lot of those are probably going to be false positives or unreachable code or low severity or maybe it's a real valid issue and there's no patch available and the team just can't do anything about it so put in that work up front to validate the

findings bring them the relevant things that they need to actually action and don't give them a lot of like busy work that they don't want to do and don't be removed from like how engineering writes into place code you know don't just like throw things over the fence and say like hey we need this thing done go do this thing without understanding the complexity or the size of the ask that you're giving them that's not going to be very effective oh and here's another tweet that we enjoyed so this is Dino ahead of security a cash app um you know if engineering doesn't like working with you you aren't going to have a secure environment no matter what

else you do I think that's a very true cool um security teams that are disengaged from the rest of the process in the same line I think um if if folks are interested in doing the right thing and they reach out to you and be like hey I'm going to do this third party integration or I'm going to have this whole new uh data source exported how do I do it right and you're not engaged and you're not telling them what is it how do they do it right and just being disengaged never works so I think the quality of your security program or how well you do from a security perspective is directly related

to how engaged you are with the rest of the program um this is another thing that really is a pet peeve of mine giving useless advice so I mentioned some of these already don't click suspicious links what's a suspicious link I have no idea don't connect to untrusted networks I think this is really outdated advice everything runs over TLS now it's just not a real issue this uh FBI Denver tweet I think went viral in the security company Community um a few weeks ago they posted this tweet saying like watch out for juice jacking attacks in airports which is not a thing that anyone needs to worry about I don't know why they decided to tweet

about this but then they got a lot of flack for it um so give people like actionable relevant advice that uh you know is up to date that's not this like outdated like performative like check the box kind of control don't give generic advice you should be specific and actionable an example of this I've seen um examples in the past where someone you know you give someone some advice like oh you're accepting file uploads like um scan that from malware and then that's it that's the like one bullet point of feedback skin items from malware it's like how's the team supposed to do that are they going to use a vendor are they going to onboard a

whole vendor to do this thing are they going to deploy some off-the-shelf solution like how do they actually do that um like you give give people advice that they can actually follow cool now that we've spoken about what security culture is what works and what doesn't work there are some things that we use to measure security culture um it's just like any other uh vulnerability that you have and you measure the age of it you can also measure security culture some things that work for us it doesn't have to be too fancy it could be just be things that you have in your spreadsheet um but uh I start off with who do I meet

with how many engineering teams are there in my organization uh do I meet with all of them how are they doing from a security perspective if I do meet with them or don't meet with them are they remediating more issues just because I meet with them um are there uh involving Security in more of their initiatives if we meet with them um things like um monocle really helps us because again we're trying to track their score uh based on the gamification over here uh to see how how does a team perform over time like are they improving in their security posture or decreasing um things like how quickly do teams uh adopt to security controls or Frameworks

that we have been introducing um again what's the correlation directly with like do you have a relationship with that team or not or are they uh looking at their monocle scores or like their security scores or anything in that sense um what's their interaction with security I think one thing we have seen over time is um teams have started including security a support point in their Sprint planning which shows that we are doing some things right um so they go look at hey let's go check out monocle and see okay how did uh do we have any action items uh for this Sprint and things in that sense uh cool so that's pretty much the slides

that we had just very briefly in a conclusion um you should definitely invest in security culture it pays dividends it's going to be the way that you scale your security program and make changes especially across a larger organization the larger that your organization is the harder it is to change culture so start early and that'll really pay off and there's no Silver Bullet you're gonna have to you know try out whatever works and doesn't work in your organization cool uh we're hiring we're hiring for a data security expert so if it's something that you're interested in you're helping you're ready to come help us secure all our data pipelines and everything in the world of data uh to

reach out to us we will open it up for questions yes

what advice can you give on that do you feel like the CEO itself is somebody who has to beardy and and does he help that talk about that coach and then to how did Netflix get to where they are because they seem to be the outlier Within yeah do you want to repeat the question oh sure um given that Netflix culture is different from many others and um I think if I'm paraphrasing the question correctly um it's to say that how does it work for organizations that don't really have a top culture uh is this um and uh What uh mechanisms can a culture a company with that culture do um I think I think it's more prevalent

for companies where there is a kind of a top-down approach instead of I think for Netflix you might not run into that accountability problem that often um one thing I've really uh realized that for uh situations where um the company thinks okay just because we've hired a security team it's their responsibility go fix all the issues that's where more of the top-down uh culture really is more applicable or effective is that you get them to be more accountable to say that we're just here to help your function deliver things more securely less about we are going to be responsible for securing all of your product does it make sense

yes

help enforce all right I'll repeat the question the question is uh how is our CSO reporting back to the CEO or is everyone that connection yes there is um there is a way of uh reporting risks back all the way back to the uh leadership and also the board um just like most of the companies I imagine and um yes I see so like we report back to all the different domain VPS but also we also report back to the CTO and the Cs uh the CEO if needed depending on what the level of the risk is like is a risk that someone is not patching or fixing or implementing I'm going to open up a risk that's going to

affect the entire company or what's the likelihood and impact of it depending on that way to report um if I could just add some concrete examples so we have an audit committee to the board that our CSO presents at quarterly we have a monthly engineering review where we're presenting metrics and that's being seen by leaders we have um our our boss the CSO he's managing upwards that's his job to try and like present those findings and make sure that the peers and like leadership are seeing the things that we think are scary and making sure that those are being actioned on um are there any other examples today there we go we also have an

we also have like a monthly engineering review that's more for engineering but all the engineering VPS directors all of them get a sense of that but that's more on the engineering side yeah uh sorry David yeah yeah

let's stay with you so the question was how do you scale uh dependency vulnerability management when there's maybe thousands of low severity findings and how do we make sure that we're not spanning spamming teams so we keep plugging David's talk from earlier today he wrote a tool called monocle and it's ingesting all of our GitHub data and presenting a dashboard to developers every repo gets a letter grade for how secure it is like you have an A plus score for security or you have an F score for security there's like a a badge that they put on their readme so that'll ingest findings from GitHub and we just kind of take github's word for what the severity of the

findings is like they'll triage those and you know they bought dependabot and some other things to do that so that's one way we make sure that depend about Auto merge PRS are enabled on all of our repos and then it's like solved for developers they don't even have to do anything um what are some other ways a lot of times there's a very high profile like uh I don't know like log4j or something for example where like when one of those happens then we're going to do a very high touch like actually actively find stuff ourselves and reach out to teams to make sure it's all patched but for the most part we just try and make sure

people are applying all of their updates in a reasonable amount of time and I think that's our time but we are going to be outside if anyone has more questions [Applause]