BSidesSF 2024 - Startup Security, 2nd Edition (Evan Johnson)

thanks for coming to my talk startup security 2.0 um this is my probably I've been attending bside since as long as I uh since the first time I moved to San Francisco back in 2015 uh this is the best conference that uh is around so uh really fortunate to be speaking here thanks to all the uh volunteers and everybody who puts it together it's a lot of work uh this is my second time speaking at bsides SF and my first time sponsoring uh where I'm a I'm the co-founder and CEO of run reveal upstairs and uh um really awesome to get to talk here so uh uh going back and starting at who am I and why am I giving this talk about

startup security I have worked at a number of startups um and I was the first security engineer at cloudflare I was the first Security hire at segment I boomeranged back to Cloud flare but I've seen that whole growth trajectory of um several companies from literally no security team up to uh post IPO post fed ramp the entire maturity curve of a security team uh I've made a lot of mistakes I've done a lot of things right and I want to share everything that I've learned and uh talk candidly about a lot of those experiences and this is version 2.0 of version 1.0 which was back in 2019 5 years ago at abset Cali this was another

great conference on the beach you can't beat this uh you can't beat that location except maybe a movie theater is pretty cool but uh I've gotten tons of people over the years who've reached out about this talk and it helped them when they first started a job as either a head of security or security engineer or or something some other type of role uh at a startup and um I think that this grid was what I put a lot of effort into and it was mostly a lot of like engineering things because that's what I was and engineer but over the years I've gotten a lot broader experiences in management and security leadership and I

want to share some of those as well um so I think version two this time around it's a lot of the same so you'll still see that grid except maybe it'll look a little better but most of the content will be the same and then I'm going to add some stuff for like career growth how to think about actually being uh successful at a startup and some meta like stuff and then um update that and avoiding problems that I've been seeing and uh just kind of give my two cents on all of that so uh yeah I guess I'll get started um I think it it worth it's worth saying like why do people work at

startups and I I think first is uh speaking like uh personally career growth I started at cloudflare when I was 24 years old and uh the first security engineer and like my first month on the job somebody was like here's 4,000 lines of C code we're releasing it next week can you make sure that it's secure and like it's insane level of responsibility that people put on your shoulders when you're a um working at a smaller company and uh that's something that you may never get the opportunity to do unless you work at a really small company so uh you get a tremendous amount of personal growth just by getting a tremendous amount of responsibility probably before you

really need it uh enjoyment you a lot of great people and you make friends you meet like-minded people and uh I think everybody works at startups for one like one thing that's in the back of everybody's mind is like I hope this thing sells for a lot of money and I make more money than I ever thought I could spend um that is less rare or that's pretty rare but it's something that uh is is a driving motivator for a lot of startup uh work I think reasons not to work at a startup uh they rarely pay very well so uh you you could definitely make money at a more money at a bigger company like

an Apple or Google or like with a Fang companies whatever uh and that compensation tends to follow you so if you join a company really early and you get paid a certain amount uh you're probably going to be underpaid for a long time as that company grows so uh compensation the end goal is to make a lot of money at a lot of startups but uh usually you're not making a lot because the company doesn't doesn't have a lot of money they're constantly raising from investors work life balance not a good reason to work at a startup uh it's chaotic it is uh can consume your life I lost a lot of weekends holidays three-day weekends at Cloud flare and uh

I can promise you that there's a million other stories like that that other people have because of security incidents because of I guess everybody deals with that in security but uh it does consume you working at a smaller company and then uh internal stability like by nature of hypergrowth they're constantly growing hiring new people reordering things people come people go it's uh it's just chaotic in startups and so uh you're not going to get a lot of stability uh working at a very small company um so the game plan to be successful at a startup it's kind of the same thing at any company but you have to speed it up a lot so uh you need to

build relationships with everybody at the company which is actually not that hard at like a company of a 100 people you can at least Meet 50 of them uh used to be a lot easier in person now it's a lot more challenging if you're working at a remote first company uh but you do need to build those relationships you need to solve problems really really fast and then uh as things change you just adapt and repeat that same process so three-step process one of those steps is go to step one and uh and that's all you need to know okay so putting yourself in the shoes of First Security hire I asked Dolly to generate me the

hottest AI startup fictitious AI startup and it came up with Robo brain and uh and this is the logo so it's your first day at the company maybe it is you dialing into zoom and you got a little care package the week before with your laptop and a nice t-shirt or a hoodie and some stickers maybe it's in person and uh you walk in and there's a big open office and exposeed brick everywhere and people are working out in open spaces and uh and you're in total shock you're like adjusting to everything and you sit there onboarding for a week you're kind of keeping to yourself meeting people talking to people uh introducing yourself uh people

are saying some things that are making you kind of nervous about what they're doing with data or certain vendors they're using and uh you're you're kind of cataloging all this in the back of your mind and then Friday comes around and you have a one one with your manager maybe it's the CTO or the head of engineering maybe it's a CEO maybe it's uh CFO or something uh some somebody is your manager and they say they turn to you and they say I really trust you to uh build our security program and all the decisions that you're going to make uh what do you think you have in store for the next six months and you just got

here you kind of hear the proverbial record scratch and and uh it like you hear the narrator say you're probably wondering how I got in this situation and uh and you're like what do I have for the next 6 months what am I going to do well I'm here to help you with exactly that um I've been in that situation and the first thing that you need to know is your first 90 days are critical so uh if you've ever had a manager who gives you a 30690 day plan uh it's there's like some people I found this guy on LinkedIn and this is his whole thing I think he's done some of the research on it but uh his whole he's

making a living out of helping people with their 30 60 90day plans and all the research around it it's probably hard to see but uh uh yeah if you've ever had a manager sit down and give you a 30 60 90day plan it's critical that uh I basically whenever I gave uh new new hires on my team a 30 60 90day plan it was more or less these three things first 30 days you're going to meet all your stakeholders make friends I used to say uh like you don't actually have to make friends but you do need to develop a working relationship with everybody who is going to be important from the kinly engineer to the people in

customer success to the pr team for when things go horribly wrong to the legal team uh to understand like your customer contracts you're going to need to develop a relationship across the board uh and then within that first 30 60 days or the the first 60 days you really need an idea of what you're going to do in the first 90 days and be making steps towards that I think that by 90 days it is critical that you've accomplished something and kind of told the world about it at the company so uh I think the a lot of the 3060 90day research is like by the end of 90 days you need to have accomplished something people who

have accomplished something at a company are significantly more likely to be successful over the long term and whether that's true or not maybe it's not true maybe it is uh I tend to believe it just from a standpoint of you're going to have to do a lot throughputs really important at startups and so starting early never hurt um so is there anything else here uh no but I think in the back of your mind to sum it up 30 days make friends 60 days you really need to be making steps towards completing something 90 days you need to have completed something uh I personally think that if you are a ceso or ahead of security or something and it's been a

long time since you've been an engineer this is like the hardest thing to do when you first show up if you don't have a team and you're not used to like getting your hands dirty uh it's really important that you kind of check your ego a little bit humble yourself roll up your sleeves and accomplish something in that first 90 days it can't be you opened a few job requisitions it's got to be something else whether it's you flip a bit enabling two- Factor authentication or something it can be small but it's got to be something and uh I think that it's a really hard thing for cesos to make that adjustment or people coming from much lar larger

organizations because um just the pace is different the expectations are different and uh and yeah you you end up having to humble yourself a little bit if you're used to things taking a year uh it's you got to fit it into 90 days um and anything else uh no as an engineer though I think it's actually sometimes a lot easier so if you're brighte and you naturally want to learn about what everybody else is doing listening and learning is it goes a long way and so it naturally lend itself up to lining up what you're going to do what you're going to accomplish um personal story at Cloud flare the first thing I did when I joined was the

co-founder uh lee Holloway walked by my desk and he was like Hey so do you know that our production database uh makes curl requests and I was like No And he like showed me all about it and he was like all of them have uh like this thing turned off and this like it was it was a nightmare uh but I had my first thing just by like listening and learning and uh and it's really important to to do that and as an engineer I think it's easier when you're used to doing the work uh as a ceso I think it's a little bit more of a culture shock so the more Junior you are sometimes the better it

is um once you finish that first thing in that first 90 days startups have a culture of like celebrating every little win no matter how small it is and it's really important that you get in the habit of doing that even if it's something like really tiny uh just giving a shout out to the people who helped you because you rarely do these things alone you're usually working with somebody giving a shout out telling the world about it and uh just building that kind of shipping culture and and showing that you're like a good person to work with I think the worst thing about security is when they're scary and people don't want to like work with the

security team anymore and so it starts with like just acting like everybody else at the company celebrating the small wins and uh uh um yeah being a good person to work with um what not to do so I think that it's there are some common traps here about what not to do in that first uh really first few years but uh I see a lot of people get into these really big problems problem spaces of like I'll just pick on one for example we're going to roll out service service off that's the first thing we're going to do like I've never seen that project go well uh so uh you really want to avoid those

problems I used to refer to them at Cloud flare is like trench warfare problems because you're kind of not really going anywhere you're you're just like working on something moving really slow and you want to avoid those as much as possible anytime you could do three things that you can call done in the same time frame that you could do one thing that is like half done you probably want to opt for the three things um you can't do that for forever of course like eventually you got to bite the bullet and take care of some big problems but um throughputs really really important I don't believe security is a spectator sport so I think that it's

really important that uh security teams roll up their sleeves and take ownership of of some problems and it's usually impossible that they can take ownership of the whole problem because like security usually doesn't own like engineering or like the Salesforce or like your SAS apps it's usually somebody else and so you've got to roll up your sleeves and take ownership of a part of that H but you have to work with other people to get to that end so uh it's not a spectator sport though and if you're expecting to like throw something over the wall just ask somebody to fix this like it's not going to happen if you want to see something get done you got

to do like half of it and help people meet you halfway um what not to do moving too slow it's better like I said it's better to do three things uh quickly rather than one thing slow and uh I used to tell my team at Cloud flare that you can do whatever you want as long as there's two rules one you you can explain why it's important in common English uh with like uh two sentences or less in language that anyone in the company will understand and then two you can actually complete it so uh service service off doesn't check that box but like enabling two factor for everybody at the company does you can go flip the bit uh and call

it a day it's easy to it's like well scoped it's uh everybody can it's easy to explain the value of that and uh you can call it a win so as long as you can do those two things you're probably in good shape um this is the Big Grid that I had from the last one and I don't love this I think that there's some things missing but 4x4 I was kind of stuck with the with the dimensions of what I was working with uh there's some notable absences but I think that it's pretty good uh I think if you compare this and the last one it's uh a lot better coverage but uh I kind of think that

most security initiatives are going to look like any Security Org chart you're going to have things in the security engineering bucket in the compliance bucket in the detection response bucket in the Enterprise security bucket as a security engineer I personally uh really latched onto the engineering side of things and it took me a long time to realize just how important the other parts of this uh this chart is and and um you really need to have a holistic approach when you're early at a company and uh not focus on one thing too much or or uh you risk like uh just not having full coverage of what you need I'll go into each one of these in some

greater depth so security engineering I usually look at it like uh like there's the normal cloud and appsx security this stuff looks pretty much the same every company there's Security reviews there's like security controls around like not making your buckets public and exposing data about not leaking AWS Keys like this stuff really is I think uh not that differentiated from company a to Company B however on the there's usually one very special thing to like each company and it's worth figuring out what that is when you're early at a company so at Cloud flare we had this giant Edge and like we're shipping servers all over the world and then booting them up and then

running our uh our our service uh when I back when I worked there and uh that edge was really unique never seen something like that maybe there's like two other companies uh with with maybe three or four with really big cdns like that so it's somewhat unique but then the way that they managed that was unique and and there's all these unique problems there so it was worth like really digging into what was unique and uh that kind of unique thing was very different than when I went to segment and this just giant data Pipeline and um and it that had very different problems and so you really need to nail what that is and then dig in a little bit there

and then not get too hung up and like get the basics done in the cloud and appc world and then uh but don't go too deep down this uh I do think that Engineers should engineer just like security is not a spectator sport you really need to dig in and and uh build something even if it's sometimes uncomfortable uh like some sometimes if you build a feature or something the team that's responsible for that area might not like it but building as an engineer even if the company doesn't always want you to just makes you understand the product makes you understand the company better uh really important not to create toil it's easy to spin up scanners it's easy to you

know install some something and then collect a bunch of results and then file them into jro or something and expect them to get fixed I really think that's not the right way to to do it especially early every bit of toil is uh going back to the same old throughputs important better to do three things fast uh it's just like things that you have to worry about and and constantly maintain so keeping uh toil as low as possible I think uh bug Bounty is a part of this like it's a delicate balance between you need a place for report like security researchers to report problems and you probably want to maintain a bug Bounty can be a lot of work so you need to be

careful uh not rolling it out too fast too big too quick otherwise it can like kind of consume all of your team's resources GRC uh not on the slide that I just remembered is uh like your company signs deals with lots of security addendums and whatnot you need to figure those out what if what what promises has your sales team made uh when you're showing up at a company um sock 2 one thing that didn't exist in 2019 when I gave the first talk is like vant andrada and secure frame weren't as big as they are today and like not every company in the world had sock 2 type 2 I think it's largely a good thing getting sock 2 type

2 early uh my company is like a year old and we have sock 2 type 2 um it's just a good practice to get the that stuff out of the way build a culture of security early so it I think it's largely a good thing but I don't think that it holds a lot of water in terms of like your security practices sales enablement I think security is a big part of sales enablement you need to create collateral you need to just like Grease the wheels so that you're not getting stuck on security when you're selling the product that that you work at or that that you're trying to sell um detection and response you need some type of

centralized logging you want it to be simple you want to have the basic logs that you want uh more important than that actually is building the kind of response Playbook you need a place for people to report problems for them uh I've used the CT at Alias where like that's the hotline you email there you'll get a response we'll semble the Jets or whatever and uh and you'll get answers or help if you need it and uh establish that early along with like how you're going to triage those who you're going to involve and uh like build out the The A Team there of the people who are going to help you when when bad things happen um but largely

it's just like fishing emails that comes into that um last is endpoint detection I'll get an EDR I think that like I'll get more into this but uh it's a big cost and um it's something that you end up having to do but uh like sometimes you don't do it immediately sometimes you will um Enterprise Security edrs on this list too kind of straddles a line but in 2024 there's no excuse not to have UB keys for and be UB key only at your company especially a startup there's not that many people that you can't ship UB Keys everywhere in the world uh or you can't like buy one for everybody at your company I promise you it might be uh

this is one of those things where it's worth it to kind of invest in do something that might be slow uh because like we've seen the last few years people are still getting popped by basic like hello I'm from your it team please go to evil.com and enter your password like it's amazing that that still works um single sign on onboarding and offboarding in the Enterprise security spaces will save you a lot of toil and uh make you friends with it um so this has more context I'll just like with some other stuff filled in I'll just pause for a second because I see a lot of phones coming out occasionally but uh I think that overall like there's

infinite depth you can go into on of these things but just scratching the surface is usually the 80% rule tells you that you'll have a fantastic security program if you just scr do these things uh you don't have to over complicate it you don't need to bring out like the these crazy Frameworks with 100 check boxes uh security is not that hard one thing that I didn't include at all in version one of this talk I think it's really important you establish your security story early on which goes holistically across everything that you're doing for your security efforts at Cloud flare this was really easy being a security company so I don't maybe have that gr of advice for

people who don't work at security companies so sorry about that but uh we we used to say we secure Cloud flare using Cloud flare so our security team is the best first best customer of all of our product managers and so we worked we like used the product ourselves and all that stuff uh and that made it really simple but you really need to establish a security story of like what controls you have for that special thing that you do at your company uh what are the things that really matter to your company and uh how have you nailed those and you should publish blogs about those you should be loud about them give conference talks about them and uh if

you find that you're not able to do that because so and so legal doesn't want to talk about it or something like you're probably not in a transparent very transparent company and uh you probably don't want to be there for very long it's probably not going to work out uh so be very intentional about establishing this security story both internally and externally and if you're a ciso that's part of the job if you're an engineer and you're not the head of security or something you can still contribute by like focusing you're you're writing those really specialized blogs about how um how you do that one special thing and those really good controls that you've built around uh the

the the thing that really matters what your assets are at that company okay uh just repeating it for the 10th time is like don't waste your time the the longer you wait to do things the harder it becomes to do them later as the company grows so there's some things that you should like calculate on the back of a napkin if we grow our headcount 100% will this be harder and if the answer is like significantly so you should probably opt to do do it sooner rather than later and um and so there's like a Time Horizon where things start to be way harder both because there's more people and because there's more like cats to herd and it

just becomes very challenging the longer you wait for some things so it's important to um to do that kind of calculus uh I also think going back to the EDR thing that I just scratched the surface of uh you need to choose your fighter carefully here I'm an engineer so whenever I see a product I'm going to walk around the Expo floor tomorrow uh at RSA and I couldn't be more excited cuz I I just like the chaos it's like a casino in there a little bit and uh uh so I'm going to walk around and I'm going to look at all these products and I'm going to be like what does this do how does this work and the engineer in

me is going to be like I could build that and uh usually I'm right I could build a worse version of that uh and like you have to avoid putting yourself into that trap of I could build that I've seen companies spend like millions in engineering resources to save tens of thousands of dollars uh and like they could have going back to the initial like the the time thing they could have bought the thing for like a 100 Grand and then been a year further than they were had they not spent a year building this like thing that they could have bought for like 50 Grand or 30 grand uh sometimes it's ridiculous these decisions they make I think some of that

comes down to uh at startups sometimes people costs get overlooked uh and so it can be easier to hire a new headcount and have them work on something that would cost much less than you're going to spend on the headcount then it is to uh like get them to say we're going to buy this vendor uh and so there's definitely some calculus that you need to do is am I going to build this or am I going to buy this and I actually think that building things that you're really proud of that are going to be differentiated uh really great thing to do I mean we did that at cloudflare uh for a little bit of both

of these reasons because it was cheaper to get headcount than it was to uh to buy venders and also because uh we we needed special things for ourselves so uh just be mindful of what you're building versus what you're buying and um yeah opt to save time unsolicited career advice for if you come from I think the hardest job to have at a startup if you're the first First Security hire is like coming from a big company and being the head of security because you're used to like oh this is going to take a year at my last job like you don't have that kind of kind of time and if if you're going to

hire people it's going to be like 6 months before people show up if everything goes right and uh if things don't go right you could be a year in and having have accomplished nothing so be very careful if you're it's a very tough job to come from big company to smaller company and be head of security if you're not don't have your head on your shoulders correctly um Engineers I think like avoid the C of like chasing shiny objects and and uh I also see get reach outs sometimes from Engineers who are like I think I should be the head of security and they might be really good engineer but going back to that breadth

of of uh like what a security team needs they probably don't have everything they need breadthwise and so there's plenty of time to like engineer now and grow into that later I don't think anybody should rush into being a head of security because also a very fun job uh and then uh what did I say for managers oh yeah sometimes managers like I think the more you know the better so if you can get your hands dirty and learn from the experience it'll make you a much better senior leader and manager because you build empathy with like the people actually doing the work all that stuff so uh if if you can get your hands

dirty even if just for a little bit um it's a good thing I am completely out of time that flew by but ending thought is I actually think that security at a smaller company where you're meeting people where they're at where you're able to learn and what's going on directly from the people doing the work usually is a lot better of a security program than at the big companies where people get so lost in the sauce of this program this risk scores that like it can be very like nobody understands why anybody's doing anything at some of these big companies uh it's very easy to get lost in the sauce of like of uh we

need to be mature and this is what big companies do but I actually think that uh the simpler the better the more you can meet people where they're at and act like you're a small company as you grow the more likely you are to be really successful over the long term so uh I'll be at the Run reveal Booth if you have questions but I'm completely out of time and really appreciate everybody uh coming out to the talk [Applause]