Concrete Steps to Create a Security Culture

um that's all welcome uh I promised I would try not to put you in this Arkady Tuttleman with concrete steps to create a security culture do you guys hear me cool so before I jump right in I just wanted to start off with a question to you all how many of you have had trouble convincing a engineering team to apply some sort of update patch some sort of software make some sort of process change you're trying to tell them this thing is important they're saying we don't care we don't want to do the thing how many people show hands have had that happen most people probably so this talk concrete steps to create a culture of

security is I think my attempt to solve this problem so specifically we're going to talk about why does it matter why does it matter to have a culture of security and what does that really mean I'm going to jump into concrete examples of things that I do at my organization that you can do at your organization that will take up the majority of the time then I want to zoom out a little bit and talk about guiding principles things that if you wanted to try and do something new at your organization what would that look like what are the things that I'm thinking about when I try to you know foster this culture of security

and then we're going to talk a little bit about measuring success and what that means and hopefully time for questions at the end really briefly about me Who am I my name is Arkady Edelman I work on building the security program at lob a small start-up before that I worked on apps a cat Airbnb and Twitter and if you want to know a fun fact about me I don't know how to put right a bike that's really true you're welcome to roast me after the talk okay so why does security culture matter well for starters I think that most security work is not actually done by the security team if you think about all of the work that's going on in

your organization I mentioned a few patching software updating libraries resolving bug bounty reports resolving pen test reports sometimes it's going to be the security work doing that team but most of the time not I think most of the time it's going to be the secure regular product or engineering teams implementing those fixes similar things for account and product security features like to fao auth sam'l etc and most of the time it's going to be other end organizations doing that work the security team has a central facilitating role to play and they'll do a lot of other important work but most of the you know fixing these vulnerabilities is usually done by somebody else and the

corollary to that is that I think that the security team has to work through influence you're asking all these other teams to do work on your behalf you're saying hey please fix this thing please match the software it's very important but you're not in their reporting chain you can't tell them to do the thing you can't say I need this done by X date you know sometimes you can but most of the time you're just trying to convince other teams to care about the things that you care about and that's very hard to do to get someone else to willingly take on work that is not you know they didn't plan for that work and this vulnerability came up and

now they need to resolve it nobody wants to do that so I think that's the first reason that security culture really matters a lot is when you do have a strong culture when people can see all the work that you're putting in and all the transparency that you're providing with you know some of the examples that I'll run through they understand that it is important and they are more willing to work with you so you have less friction and less resistance and pushback from these teams when you ask them to do something important the second reason I think is employees are really your eyes and ears on the ground there's gonna be all sorts of

things going on all the time that you just don't have visibility into I've had employees come to me with things from you know there's a password in a Google Doc to someone sent out an email with a sensitive PII information to someone's trying to build a new feature a new product and one of their peers will say hey did this run through security review can you get this reviewed right you want people to volunteer information to you not to force them or install gates or processes like it's better if people just fall into your information to you and when you have a strong culture of security they will do that next up inspire people to not bypass controls it would be

really great if all of the controls that you put into place we're just rock-solid you can't get around them that's not always going to be the case a lot of times when you introduce some sort of friction which is unfortunate when you have to but when you introduce that friction people often try and find a way to work around it because it breaks their workflow somehow it like introduces inconvenience for them and if you are regularly sending out these updates and showing people why it's important they're not gonna do that hopefully and the last one is build social capital I mentioned that not all security updates are going to be transparent and invisible there's a lot of times where

you're gonna introduce some sort of breaking change or there's going to be some sort of trade off you know security is all about trade-offs and so you're gonna want to introduce some change and introduce like as an example you want to enforce two FA you want all employees to like only be able to log in through two FA and some functions and your company probably don't want to do that they don't want to deal with that they might not understand why it's important or they don't like that it takes longer to login and when you are very visible with your work and sending out updates all the time and doing these other examples you're building social capital that you

can later use you can spend it as if it's currency to implement the changes that you want to implement within your organisation so those are the reasons why I think it matters now I just want to run into eight concrete examples of things that I do have my organization first one is security champions I think this is really common that's why I wanted to start with it a lot of companies do this there's many different versions of this depends on the organization at larger organizations you'll often see security teams embedded with an within a product team and they're gonna be the go-to person for any security questions there if they have to escalate to the central security

team they'll do that that works well at larger companies my company is pretty small we're only 70 people so our security programs security champions program is literally just a Google Group it's a security and Interest security enthusiasts interest group and I'll just send out discussion topics things that I think might be interesting for people that they might get value out of and we'll kind of discuss it it doesn't have to be a big heavy thing it's like a very lightweight and I just email a few lines every now and then so security champions pretty popular this next one I call fireside bug bounty as my number one favorite thing it's my favorite thing that I'm going to talk about in my

slides and for those of you who run a bug bounty program when you receive a valid report that you triage and it's you know a real report I like to send out a detailed write-up of what that report was and what the impact was so the text of this email doesn't really matter if you can't read it I just wanted to include an example picture but I sent out an email that said hey here's this issue that we saw and here were the implications of that and I'll always follow the exact same formula for these also with a description of what the bug is how to exploit it I'll include a proof-of-concept if applicable you know

here's what the payload looks like here's what the request looks like I'll talk about the impact of the bug what could someone do if they had exploited this what data would they have access to what else might they have access to I'll talk about fixing this particular instance and I'll always give props to the person who did fix the bug I'll link their pull request or their code if applicable I'll say you know if they did a particularly good job and solved it quickly I'll call that out it's really important to appreciate people's work and give them that positive reinforcement and then I like to talk about what would it mean to solve this entire class of vulnerability when

applicable so the thing that I love about fireside bug bounty is I think a lot of developers are actually quite naturally curious and want to know about security and they just don't know how to get started they don't know how to get into it so when you make content like this available they eat it up people respond very very well to this and are much more willing to work with you when you are regularly putting out like here are the vulnerabilities that we're seeing here's how we fix them and here's how we might fix this entire class of vulnerability and for that last bullet point in particular I think it's important because a lot of times fixing a class of

vulnerability might take multiple quarters or even years of work if you have two riryoku tech major system components and if you tell people about it early enough in these emails then you just put it on their radar you kind of plant this seed in their head that this is something the security team is thinking about this is something that's important to them and maybe six months down the line that component happens to be going through a rewrite anyway if you came to that design meeting and you said hey let's do this thing because it prevents this bug class and that's the first time they've ever heard about it they're not gonna want to do that it's a

lot of extra work they haven't heard this thing before but if you have been talking about it regularly if you brought it up in these past emails if they are already aware of this issue then I think they're gonna be much more willing to work with you third example again the emails don't really matter here I just wanted to include some pictures so send out current happenings when browser started enforcing certificate transparency I sent out an email about that even though it's not super relevant for our organization when we deployed pre-loaded strict Transport Security on our website I sent out an email about that and we had a quarters worth of bug bounty data I sent out an email about

that how many bugs have we resolved how much money have we paid out how responsive how responsive are we how quickly do we deal with these issues I think people really enjoy seeing these numbers and these statistics or you know some of the other emails in general and when they can see that you're like working very hard and you're open about the work that you're doing again they're gonna it'll put security top of mind for them they're gonna include it in their considerations when they're doing other decisions in their day-to-day work so yeah I just write up current security events and explain their significance make sure you know your audience if you're working with so lob is mainly a

web application security we don't do we don't have a mobile app we don't do anything else complicated so mostly just regular back-end engineers and some of them might not be familiar with certificate transparency for example so make sure you know your audience when you're explaining your topics so it's approachable for them hacking 101 our company had an off-site to Palm Springs and we were doing all these exercises and workshops they were trying to train people how to be a manager how to like I don't know all these other things and so I thought wouldn't it be fun to teach people some basic hacking techniques and if you're familiar with a wasp reduce shop it's a intentionally vulnerable web

application that people can learn how to hack against it's not the only one there's others but I like this one so I taught everyone how to set it up on Heroku they each had their own personal instance I showed people how to install burp suite and configure it correctly actually sent out really detailed instructions I've like literally step by step like click these buttons and enter these configuration settings because I wanted to handhold the developers a little bit and make sure that they had a good experience I think if somebody tried to like if I sent out this thing and they couldn't get the pre work done they might just get discouraged and give up and I think if you you know guide

them a little bit then they'll have a great experience and so this I think aside from the fact that developers enjoyed it it also makes them better coders right if they see how to attack a cross-site scripting or an insecure direct object reference or something like that in somebody else's code then the next time that they're writing a piece of code they're gonna think back to that and say oh I better check that the requesting user actually owns this object that they want to modify and I say postcards this is one of my favorite things that I did what happened is in 2016 there was a government watchdog group that sent the Freedom of Information Act request to the NSA for

all of their security propaganda posters from 1950s and 60s kind of like you know loose lips sink ships if you've ever seen that one before and it took them two years but in 2018 the NSA responded they gave these hundred and thirty six posters and to give you a sample of what they look like here's one it's some college graduates and their plans for tomorrow depend on security today or another one is you know it's a light bulb and it says let security light up your life it's a little bit cheesy but so LOB's main product is a print and Mail API you can use this to print postcards letters and checks so I used

our own API to print all of these different postcards and mail them to our own office and then I handed them out to different employees and so that was really fun I think people got a big kick out of that and also I enjoyed this because all of the previous examples that I gave were really engineering centric they focused solely on engineering and this one I think everyone got to enjoy this talk was supposed to be about things that are easily replicable I assume that most of you don't work at a print and Mail start-up so I don't know that you could do this but the idea is to get creative and maybe use your own product or come

up with something that is gonna be relevant for your organization that you know that your employees are going to enjoy security conference watch party I just set up a meeting every two weeks we all get together at lunch and watch a security talk together it's pretty easy I send out a reminder of the day before with the summary of the talk and the speaker which means that I never will select a talk that I have not yet already watched I try to curate content that I think will be good for people that's not too long if it's you know an hour or something I think people get a little bit bored by the end so I try to

do 30-minute talks or less not too technical something that I know again is going to be appropriate for the audience and I'll try to mix up the content so for us the thing that is most relevant is web application security and a lot of times it is a web application security talk but sometimes you know one of the most popular talks that we had in terms of attendance was a video game console aking talk or a car hacking talk so that's not something that we're ever gonna use in our day-to-day life but people thought it was interesting and they attended the talk I'll send out my slides later these are just if you want to use them these are the talks that

we've watched so far show-and-tell we have this tradition at LOB this wasn't even set up by me when I joined the company they already had this every other week on Friday at the end of the day it's five-minute lightning talk presentations by any employee on any topic so I love to use this as an opportunity to share some sort of security knowledge with folks and every talk gets a technicality rating from one two three one is least technical three is most technical so I'll try to present different things that I think might be valuable to people I present it on things like should I use a VPN like what does it actually protect me from and what does it not protect me

from and what are the trade-offs or chrome site isolation or other things like that again not really relevant for a day to day work but I just want to be putting stuff out there that is personally interesting to me that I think other people will also enjoy and I do think it reduces a lot like I mentioned in the in the Y section it reduces the amount of friction that I get working with people and then the last one I wanted to mention security little wins I send out a monthly email calling out all of the small security that wins that we had over the previous month so this is an example email I know

it's heavily redacted but it'll just include the person's name at the beginning because again I want to celebrate their win and show them that I appreciate them it'll include a line for a short description of what it is that they fixed and maybe like a link to the JIRA or the poll requester what have you I had more things but there's not enough time I wanted to talk about a few things that didn't work two things in particular one is security percent twenty percent time if you're not familiar a lot of companies have this concept of twenty percent time where you can work on something that is not your main project it's just a little side

project maybe somehow still relevant to work so what I thought I would do is I would look through our backlog of tickets find ones that were easy starter tasks not too much work that were good candidates for 20% time and I would tag them and send out emails to people saying hey if you're looking for a security 20% time Talk project here are something that you can work on and in my time there I've not had a single person take up a security 20% time task and my theory is that people are already tired from work if they want to work on some sort of side projects they have their own side project in mind

they don't want to work on someone else's side project maybe I don't know if you have more success with this please let me know but I never got anyone to fix one of these tasks another thing that didn't work too well I think I didn't focus enough on non engineering functions you saw that most of my content was targeted towards engineers and that's probably where most of my interactions are day to day but at the company as I've started to work on more things with outside functions like accounting marketing finance HR etc I have noticed that I do get a lot of friction from them because they're not seeing any of this work that I'm putting

out they just see me show up to them and start making requests and they don't like that so guiding principles like I mentioned if we zoom out and think about like what are kind of the common themes between all of these examples that I presented I think I would call out just three the first one is add value and when I say add value I don't mean run a bug bounty program or like prevent RCEs because that's just your normal that's your job I mean add value to individuals teach them something new put out informational and educational material that they'll enjoy reading and they'll want to come back to you for more be visible I'm constantly sending out

emails updates slack messages some companies do security scorecards I think that works really well there's this great quote and I wish I knew who to attribute it to but it goes something like if you want an employee to internalize something you have to repeat it so many times that you become sick of saying it and that is when they will start to hear it and I think that is very true it takes people a lot of repeating to you know people are busy they have their own things going on so I don't begrudge them for that I think I'm the same way probably everyone's the same way but if you do want to get your

point across you can be very visible and constantly sending out these updates and the last one is but the relationship first so be available to people when they need help this you know depending on your organization this could be email or slack some organizations do security office hours I think that works pretty well security is a service role I think so you should think of employees as your customers and try to have good interactions with them even if it sucks when the answer is no but if you do have to say no the way that you say no matters a lot and also this isn't just for work stuff this isn't just for you know I need a security review can you

look at this design document I have people come to me with questions like my email got hacked or how do I secure my accounts or I read this article about phone porting attacks I don't know what to do and I love it when people come to me with questions like that I actively encourage them to come to me with those questions because I mean I do genuinely enjoy helping people but also I think that again it helps build that trust it helps me work with them later on when I want to implement some sort of other thing and they know that I'm coming from a genuine place that I really do think this is going to improve the state of

security somehow so I started this talk with widest security culture matter and I it wasn't a particularly controversial position I think that if I told everyone in this room that you could just snap your fingers and your company would have a culture of security nobody would ever say no to that the problem is that you all have other things to work on you're all very busy people so the better question is how much should i prioritize security culture over my other work and that is a much harder question to answer because that requires measuring the ROI of all of that work that you're putting in and how do you measure the ROI of culture right that's very hard so measuring

culture is hard and subjective I've tried a few things here I've tried surveys it works ok I'm still trying to experiment with what might work best if you have ideas I would love to talk to you about it I am anecdotally seeing improvements especially in engineering again I think it's easier to work with them now than it was in the past but there's many factors to that and it's also very hard to answer the counterfactual question of if I had not spent that time working on these things what would my interaction have gone like with that person would it have been worse or better I will say that I think you should invest more at the beginning than at the

end because culture has momentum if you invest a lot of time at the beginning and setup the right processes and new employees are coming in and their peers are telling them like yes make sure to get that security review culture has this momentum and it'll just kind of keep going so you invest a lot at the beginning and after that you can kind of go on cruise control but if you wait until it's too late if you have a very large organization and you start trying to do this it's going to be much harder it's very hard to change the culture of a large organization so I wanted to end this presentation with this tweet that I

saw it says InfoSec nobody listens to security people also InfoSec your password needs to be at least 15 characters long contain numbers upper and lowercase letters at least four special symbols you can't write it down and you have to change it every 90 days so I think there's been a real shift in the industry where people I'm seeing a lot more tweets like this and talks like this one where people are emphasizing this point that it is a service role you have to be an enabling function for the business it's not yes but it's yes and and help people do the things that they want to do safely and I think those things are all true but I think nobody

talks about the point that there are a lot of people who work very hard to do that probably many security practitioners in this room who provide reasonable trade-offs modern advice they compromise with people they meet people halfway every day and it's still an uphill battle for them it's still very very hard so I think my goal with this talk was to say that it's probably still going to be an uphill battle but when you have a strong culture of security it's gonna be less steep you're gonna have less friction and resistance and pushback when you try to work with these other functions and I think as an industry if we keep moving in this direction we are I hope gonna get to the

top of that hill if you're interested in my tweets there's my Twitter also I'm hiring if any of the things that I said resonated with you please check out that bitly and with that I'll take questions [Applause]

our first question from slide o is do current happenings go to the whole company your security chain pinions or something else sorry could you repeat the question do your current happenings go to the whole company or your security champions or something else great question and it depends they mostly go to security champions things like certificate transparency and strict Transport Security pre-loading I sent that out only to security champions sometimes there's things that I think are relevant for the whole company like back in like September or October Congress passed a law that made it free the credit agencies were required to provide free credit freezes and before that they charged like a $10 fee so when

that happened I sent out the entire company a note that said hey you can freeze your credit for free you should go do this thing next question from slide oh is when doing a hacking 101 class how do you prevent the few security enthusiasts from taking the spotlight for me it was pretty easy because I'm the only security person at the company but if you were in a larger company that had more security people then yeah you have to take classroom dynamics into consideration I don't know if I have a great answer for that okay and when will your slides be available when will the slides be available I can publish them probably right after I get

out of here I'll put them I'll tweet them out and I'll put them on my website and lastly who is that tweet by who was that tweet oh I mean I intentionally censored the name I wasn't trying to like throw shade at this person I do think I follow that person I think they have good tweets I just wanted to take a point on that tweet so I'd rather not say okay cool are there any other questions in the room great then on behalf of Eastside San Francisco I would like to thank you for your presentation today [Applause]