Titans of Scale: Strategies to Scale Security in Expanding Organizations

I guess everyone is here to listen to the titans of scale so let's give them a round of

applause and I'll pass the word to our panel moderator Ariel Shin who will present each panelist one by one welcome everyone to today's panel Titans of scale strategies to scale security and expanding organizations my name is Aon I'm a product security engineering manager twio and the goal of today's panel is to share real world experiences Solutions and hard-learned lessons when it comes to scaling security as companies grow so to does their technical debt product features and exposure to threats an effective security team must adapt and solve scale its people's processes and tools uh we have a great diversity of panelists today who have effectively scared security at organizations of varying sizes and growth we'll dive into how

these leaders think about scaling when it comes to security teams tooling and programs the format of this is the we have a slido I believe not desire to show the instructions are but besides sf.org qna so please add your questions in there about anything about scale and I'll be looking through that throughout this pres uh throughout this panel and then uh we have some pre-prepared questions so first up I'm going to ask everyone to introduce themselves and what is their favorite technical book we'll start with Julia I'm starting my name is Julia connect pronounced she her I work at Netflix I lead our security platforms engineering team there uh my favorite technical book so I am a manager now um and so my

favorite technical management book um is uh been thinking a lot about an elegant system by Will Larson a great book um yes hey everyone um M here I lead our product security team at ch um favorite technical book I think I would go with uh building reliable and secure systems by the Google Sr Team U I think that's a real solid book for anyone getting into the space of building um systems and just how you go ahead by securing them uh I'm Jim Singh uh a longtime developer and then longtime security engineer I'm running the product security team at ripling I haven't read in a long time uh most I have busy with my kids stuff but uh I

think the last one that I actually enjoyed was Money Ball the book by Michael Lewis um made me really think about metrics in a different way so it's nice to just sort of pick apart the different type of metrics that we're using I'm Jacob Cassie been on IRC since 1992 uh been leading the snowflake been leading fnet still on fnet anyway been leading the snowflake security pro security program rather for The Last 5 Years uh and my favorite technical book is write gray code volumes 1 and two not three okay so for our first question for everyone uh we'll start with Jacob for this one what is one challenge you've come across with scaling at a large

organization that audiences may not have considered um so I think the answer for what it means to scale over time uh something that I learned or or maybe the a question a hypothetical question cuz there's too many people in here to throw out an answer but like it might be intuitive that the difference between supporting 10 and 100 developers is different or 100 to 1,000 developers is different but that the difference between 1,000 and 2,000 and 2,000 and 3,000 um is quite different it becomes more complex and there are more problems that you need to deal with and what I found was that I did not anticipate those particular problems and what you also learn at scale is that the things

you fail to anticipate come with an increasingly high cost each time you fail to anticipate them um and then maybe the other thing I'll mention is that often your fate is going to be coupled with the maturity of the overall engineering organization their developer practices and the practices of the infrastructure teams that you're ultimately supporting especially when you want to get opinionated or rather reduce the amount of choice that's available all right I guess I'm up next uh I think a I'll give a specific example for this um when we were working at too um Ariel was actually on the team we rolled out a different vulnerability Management program um so it's completely different from um what we were doing

before and rolling it out for a couple hundred people no problem we send them slack messages uh emails people read it when you roll a program out for several thousand people no one's going to read their emails no one reads their emails no one reads their slacks so it's way harder to sort of figure out how do you actually roll out a program to multiple thousand of people so um area did a great job where they went on road shows and went to every single business unit and sort of explained what we're doing but like when you're comparing small scale to large scale um you have a huge problem in communication so that's always an interesting problem to

solve yeah I think there's been a lot of talk about like building a lot of developer tools shift left all of those work but I think the biggest challenge you will face is do you still have that collaborative culture with your engineering function and if you do not have that that's where your biggest challenges are going to be so um it's actually just meeting with all the business units the leaders there the managers there the engineers there having those conversations keeping those conversations running making sure that is a collaborative nature um that's actually what actually matters in the end of the day um I mean sure your tools and everything else is going to make

things easier but um um are you in talking terms with your different engineering teams and the rest of the functions is kind of what makes a difference yeah so at the end everyone has already said the answers but uh I think um from uh my perspective and maybe a little bit in summary is I think like people are still people at the end of the day that's very hard to scale um Communications relationships are hard to scale um but you're also like trading a set of problem for a different set of problems and those you're like uh maybe envisioning this world in which you're like we've automated everything and it runs beautifully and then you're like

but we don't have security Ops anymore we have engineering Ops and are we ready to uh pick up that challenge because it's a different skill set it's a different set of people who might be doing that work um and so yeah I guess I would say like are you anticipating some of the problems um because they're going to be different and are you able to kind of like predict them enough to build out your teams for them we want toore that underscore that we're really amongst Titans in the security industry with Netflix chime twilly and snowflake I've heard just really great things about all Rippling ring my my bad betrayed betrayed but um I Heard really great

things about all of your programs I'd love to hear about a specific program that you felt had failed and why let's start in the middle I I can start um so we did a fantastic program at segment where we actually rolled out self-service threat modeling um so we taught all the developers how to threat model no problem I trained 80% of the org 150 200 Engineers pretty easy to scale that um when I was looking at rolling it out at the toio scale um it was very problematic how do you train several thousand engineers and I was trying to boil the ocean I was trying to think about how do I actually do it across

everything and I sort of failed even at that um towards the end of my tenure I think we came up with a good scenario where we decided not to focus on training not not singularly on training we want to really focus on our security Champions program and with the security Champions program our goal is that we'll have uh an agenda and walk them through a bunch of trainings and once they sort of get certified as a security Champion um we'll get them to deliver the training so we can scale in that fashion but uh I think my biggest problem was that uh I was still thinking um about how do you scale think things that are

very difficult to scale I hate I hate computer-based training U it's just so impersonal it's really hard to learn as part of that I definitely want to continue with instructor Leed training and just run into problems that way um for us I think we did we we got really successful with like Services service o at time um like we give a pretty platform level solution for people to use and other developers loved it uh we try to expand that for other levels of a like member authorization and things in that sense and I think we didn't listen enough to what really the problems were in the other spectrums of what how authorization works so member

authorization is completely different from how Services service authorization Works versus employee authorization and trying to force a solution that necessarily didn't work for the Developers for their use cases um didn't really scale well for us I think um even though the same solution or the same platform did work for one of the use cases it doesn't mean that it will work for all um and we just have to like be pragmatic and listen to what really the challenges are before you actually um I know it sounds silly and stupid but we thought we understood it and that's why we built it out but in reality that's not how we um the reality of this system was

so you want to be last Julia you want to be last no you don't want to be last that's that's fair that's fair I think um one thing that was a particular challenge is kind of rolling out again I think maybe both of you mentioned like uh uh anticipating things or uh the human element right like I think that that is really hard and when we try to scale things we think a lot about like how are we going to engineer this thing but not how is it going to land on people um and uh security is um maturing a lot where like software engineering has had a little bit more time and structure so like

people like product managers uh marketing managers like these different roles that really make sure that these things roll out well and that they're communicated well where we're just not like we're cutting off your access because we didn't think you needed it anymore or something like that um and so um too specific spefic experience I think rolling out kind of um security guidance at scale with enough context to give people kind of this selfs serve um experience like that turned into like operational bunch of questions and uh uh things that then my Engineers had to like respond to in in slack getting back to people because we hadn't maybe done enough of that work upfront to be like

let's understand who we're talking to what we're trying to get them to do and is it clear enough um because you are than hitting a whole bunch of people with a bunch of things all at once and so then it comes back at you right um so so doing the extra leg work UPF front making sure you have the right people in the right places to do that work UPF front um so that you can kind of um anticipate what that's going to be like well I just thank you for that because one thing all plus one is that you have to treat especially at scale everything you do in your security team as a

product and if you're building Production Services which I'll go ahead and argue everything you build build whether it's in production or whether it faces the developers in production you need to have the right people in the room so I won't use that example so thank you uh but what I will talk about is maybe like a notable failure for us uh at least over the five years I think everyone here has heard a talk about secur defaults at some point in their life and is probably real hyped on secure defaults I'll just tell you we never got hyped on secure defaults and never figured out how to do that at scale we never figured out how to write

static analysis rules at scale that were meaningful because fundamentally one there's no database OAS top 10 so there was like the domain we were in that made certain things challenging and then two from my earlier answer I talked about engineering your fate being tied to what engineering does and uh in our early days snowflake allowed a lot of choice and in the face of sprawling Choice it is very difficult to express invariants or opinions of any kind and so it has actually taken us a very long time not just as a security team but as an organization to get our arms around the sprawl because it's a business imperative and we've now started to be

able to ship some things like mukun is talking about which we would consider secure defaults like service to service o why we built another Vault I'll get into that another time but like a a a high performance Secrets engine uh that runs in production and so I'll say toward the toward the tail end of things we were getting successful and sort of had a vision for what we could build but really struggled to do anything on that front for the majority of of the time that we've been working on it good answers uh thinking about squall in a when you scale application security for Enterprises do you believe in a centralized security model where control

is Consolidated or decentralized model that allows business units autonomy how do you balance between the two it's your turn I think this is a tricky one because um you got to keep accountability where it belongs like I that's fundamentally how you can scale the program if you are accountable for their security that's that's game over for you right there so um keeping accountability where it belongs is the hardest part and that's where um the Crux of the problem is and this can be solved by both ways a time we use a mixed approach we have a centralized security team that's building things for the rest of the company and the developers to use um and we also have

like embedded security Engineers that are part of different teams helping out the different embeded functions with their like specific um domain problems for example example we have a data security engineer like uh Jacob mentioned there isn't like a top 10 for data problem so we have a data security engineer who just like working with the data teams to figure out how do we do security for the data domain um similarly other embeded Engineers on the team but they are working with our centralized team to make sure that we this the bigger security team is building um solutions that work for the majority of the use cases that's kind of how the model works today we love it

depends um I I think uh I I think it depends I I think that there has to be some things that are Central I think to get some consistency for the uh information that you need to know to operate as a business and to know kind of as the ciso or as kind of the head of security what's going on in the different business units but there's a lot of flexibility that comes with being embedded in a business unit um I was on one of those teams and there was just a bunch of freedom to do all kinds of things but um uh and we got to kind of invent things and go really far because we had a lot of

context on what that business was operating like um what were the the things that they cared about how were they um building products and how can we then go like embed in their processes and systems and and um make meaningful change um but I will also say that if you are not on a Central Security team if your engineering leader at some point decides that security is not their top PRI priority and you don't have kind of that uh support uh then that that does become very hard and so I do think that um I like I like a hybrid model um but I also think that like there has to be kind of like these strong dotted lines

at the very least um and then whoever is if you if you are an embedded team that truly reports up through different vertical like there's to be really strong relationships uh to the security team to make that successful in the long run yeah okay well my answer was was already said it's it's it's hybrid so maybe I'll just elaborate on what at least the mental model um however you view it maybe you've seen Fight Club maybe you haven't but I think what was what what was kind of key about Fight Club is that all of the people in all of the places that you regularly visited that were part of your life across your whole life

were members of a club and they all had a secret goal so you could say that the approach we took was to have what I folks who've worked with me heard describe control both pieces of the chessboard and all of the pieces so we have you know a product sec I think in a product company you're not going to be in good shape if you have a centralized security team because you the product will I don't want to make like extreme assumptions here but like a lot of product teams won't accept like contributions to the codebase from outside of product there's a lot of reasons why that might be a good idea as a matter of fact and so fundamentally

being able to influence as a one being able to contribute code that's a basic thing but two security is really about psychology and tribalism is a very real thing and it is better to belong to the tribe you want to influence than to be like an Invader uh and it's even better if you are both a member of the tribe and you control the Invaders so so I think that's kind of the approach we took and now it's dangerous because there's a difference between like manipulation and influence and so I'll give you the the sort of answer that we use is like what's best for snowflake it maybe sounds corny but that's you know try to engage with that process

genuinely and do what's best best for for for the other person and for the business and it works pretty well um but I do find that as Julia mentioned if you don't have a partner so for this hybrid model you're paying kind of like a virtual team tax and I think it's a worth it tax because you have accountability on both sides and you have accountability with the business in engineering but as she mentioned if you don't have that if you don't have a Central Security team then you just have the fox guarding the hen house and if there's no one to protect that type of situation other words product has a perverse incentive to coers the security

person to move faster uh I think that's a really bad model actually uh Jacob I don't think you're all to speak about Fight Club sorry sorry bad offs SEC um I think everyone's right it depends um but uh the companies that I've worked for I feel that um having a bed or a democratized program has worked really well in itself so really making sure that um the security team's role is not as to provide all of the security knowledge to the engineering teams I I don't want that um this from that standpoint the Central Security program is dead um I really want to make sure that we U we allow our actual Partners or risk owners Engineering in

particular to make their decisions but we're there to support them and make sure that they're uh trending in the right direction doesn't necessarily mean that um they get to have the final call if there's something that's line incent we will absolutely make sure that they don't cross over the line sign but I do want them to make the decisions because they're much deeper into their areas and they understand uh those problems so definitely working at companies like too where they have a lot of Acquisitions it made a lot more sense there and then now working at Rippling I think we have 26 products or even more um I I do want to make sure that I Empower uh engineering

leadership and leaders there to make the decision that we're there to hold them accountable for doing the right things when building and structuring application security teams do you hire software Engineers or security Engineers more often maybe a hybrid let's hear more I think back to you now okay uh So lately uh it's software Engineers um so the the end to that question is will software eat security and the answer is yes but like there's a limit to that and where the limit lies is I think what what I learned was that you have to respect software engineering for what it is and not ask security Engineers to do that and put them in a tough spot where

you're expecting them to know software engineering and they may not uh and there's a second component of that which is that risk decomposition is not the same thing as software engineering and so if you really had to think about it I think you end up with security Architects and software engineers and what security Architects are really like our risk product owners and that there is an art to being a product owner so don't you know belittle your PM friends there's something to that uh but I think the whole risk decomposition process and understanding what is the control architecture we're trying to achieve is something that needs highlevel experts to do and they need to be paired with

engineering teams who can bring that into the business you said it way better than I was going to say it um so I I have been hiring a lot more security Engineers that have software backgrounds Security will never grow as fast as engineering and the only way that we can actually keep up is actually building out uh tooling uh scaling our Frameworks building that pave path so that we can keep up with them in general um having said that I definitely noticed that there's a gap with uh the skill set there and um to Jacob's Point um those product managers or those skills at uh TPM or someone that's really good at strategy um is not as apparent uh with

folks that come with a software background so uh I I definitely we'll be hiring more on the software engineering security engineering background but I do see uh a place for folks that come with strategy know how to roll out large programs know how to hold uh their stakeholders accountable so typically I think I will have probably a 1 to four ratio where four people uh with software security engineering backgrounds and one person at least on the strategy side yes yeah I think like similar to maybe what I had mentioned earlier I think kind of pairing software engineering with uh somebody who is defining a problem and helping to define a solution um so that the people who are building software can

be really good at building software and really good at scaling that um to solve the problem that's well defined um for them and on that team um and so yeah I agree that there's um kind of a a hybrid but we need people who are very good at um building software where I think previously it was like can you can you code a little can you script a little um but now it is like we need we need people who are software engineers and and to pair them with the problem I think it's depending on what the function is um so for me it's really like okay do I need more people to go on

the offensive side look for issues right now and then I'll probably hire more security Focus based people um I don't want to say that I do have the privilege of like experimenting a little in this area I think we've hired like Sr Engineers to come and help us with like infrastructure security uh we've had data Engineers to come and help us but they get a security problem but I think I was belittling the amount of effort it'll take to onboard them into the security domain um I think I was initially under the assumption that it's going to be easy to teach them security stuff but like you can't even talk to them about like go look up the oop and

they'll be like what's that right if like there's a lot of assumptions being made that it's easier to teach someone that security stuff um so I think if you do have the privilege and if you do have a supportive team that will help you like on board these people into the different domains that's the right thing to do because the security space is just too small um everyone's behind the right because the best tent so how does the toxicity of certain team members or leaders affect the scalability of Security Solutions in a growing or can you share strategies or man for managing such challenges while maintaining security efficiency spicy I went straight forward I was going to add a little intro but

decided to just go for the kill okay so how does the toxicity of leaders and ic's Y team leaders or leaders or sorry team members or leaders yeah I I really like to try working with people first um and understanding their viewpoint on why we can't roll out certain programs or um software automations and we'll run into problems uh we'll absolutely run into problems we'll find parts of the ecosystem that just doesn't want to get along and I I think some of the strategies that I've used is like first try working with them um get Buy in from them um no okay let's go to um we can either escalate on their side and um go

to their managers or escalate on my side um but like I have to make sure that I jump through the proper Hoops I have to make sure that the programs that I'm trying to running will actually work I don't want to go through escalation only find out this not going to work and then my street credit is going to go down further so I do want to make sure that I have the metrics and the data that I need in order to fight this particular battle but um a lot of times uh after escalating and making sure that uh all parties are on board um we'll have a decision sometimes um I come up up front

and it's good um we move for forward sometimes they need more data and understanding why we really want to push forward with this program and I'll try working through it um but there are times where nothing will happen and that leads to anger frustration um just uncertainty on uh Team side and if there is too many instances of that happening I have left companies for it why hire me if I can't roll out Security Programs that are going to make large impacts I rather go to places where that can go out and just roll things out and get things done spicy answer I want names I think I need to get paid a lot more say those

things what's what next me yeah you okay um I actually love those kind of scenarios to be honest I think those are the most fun problems to work on uh and I think that's you you kind of realize where what breaks in your own program and what breaks in your own pitch like when you're reaching out to people um and that actually helps me like go back and redefine on how I should be selling this um also try to understand why they are so sore right like it's maybe because they just had all the previous jobs they worked on had really shitty security teams and they just don't trust security folks anymore um so I like try

to take like like I'm like okay like let's not talk about security like let's talk person to person like just understand what is that you're turn on and then have that relationship be built on um but yeah there have been times even when none of that has worked and then yeah escalation reaching out to my boss to talk to their boss um all of those are definitely scenarios that have helped um and actually uh this is where you get in that buy in from your uh sea level to say that do they really think security is a priority and if they do and if that's what they really cared and this is a scenario where they prove that

yeah that this is a priority for them and they handle the situation there's so much assuming good intent here yeah um so so you say toxic um I think I think that this is an interesting question because is it yeah I think like uh juven and Mund are doing the thing of assuming good intent which is good and we should and so I do think like I agree going to them and saying can I understand what it is that you want um so that I can then figure out how to let you know how what I want might align with that outcome or how I can help you get there I think that's kind of like

the primary thing if they are kind of indeed a well-intentioned person who maybe is just being stubborn um or or standing in the way um I think yeah maybe true toxicity is is really difficult and it depends on kind of where they sit and uh what they're doing but but I agree like trying trying to work with them and um figuring out how you can align goals and understand theirs help them understand um yours I really do try to connect on like a human level and be like I am trying to do security here and here's the reason why it's important for the company like what Jacob mentioned earlier we say that a lot of Netflix um do what's best for

Netflix and that is usually something um that you can kind of connect on first and then build backwards into like how do we do that together um and and why am I trying to influence something in on your side and trying to connect on that as well so I'll maybe say talk about two themes that that I've heard and of course you you've heard the part about like the Happy path where you go and engage with them and in common interest in those things um and so Jen mentioned one thing that I think about a lot which is uh like toxic people are everywhere that's a fact of life especially at scale uh bad intent is is true at scale

like lots of other things are true a skill that you don't necessarily plan for um so the thing I think about the most is how to protect my team from toxic people so let's talk about toxic I's like my message to my team some of them are sitting here so you can tell me if it's true or not but like we don't have time for that you do plan a and and then it's like okay we shut it down because what it does is what Jen described your team quits and burns out because every day they're arguing with with someone who really doesn't have authority to make this argument so again I want to just reiterate you need to do

all the nice stuff first so I'm just going to give the the other opinion on this in in the SE Suite if you're that's a different thing right and if you're in a security team who doesn't have seite support like rip I'll pour one out for you but it's not going to be good but in this case it's like this is what I talk about kind of the fight club thing which is great you know like we have a risk here uh we're going to be very sure that our risk and we have the metrics as Jen mentioned and all those things but like okay let's go explain to the board why we're not why we're not you know what I

mean like that never happens so you just the the thing is whether or not you kind of have the courage to do it people don't like conflict so security is often about conflict we may describe it as other things but like I's let their managers deal with it you know what I mean like do the right thing and then let the engineering Orcs deal with with their toxic ic's make it about those leaders and holding them accountable for not having a toxic org and for not having people or rather an organization that's not taking security seriously that's where I would rather Focus my energy but overall managing up has a cost and you really have to manage that cost

because as an individual it will wear on you and on your team it will eat them and Destroy them in most cases so so we've heard some stories in which Engineers were bad Partners to security and preventing scale but are there any instances in which security was the real bottleneck or the toxic member and the performance of a product when run at scale any recommendations on how to navigate this space you're up for never heard of it I don't want to lose my job so um yeah I think there are like uh in my current place and previous places there a lot of like teams within security work completely different like I think my team is very transparent and shares

everything and we're like if you can't if you don't share with people then I'm going to know what to fix versus there are other teams that are more like let's make sure we don't if we do get into an issue we don't um have a lot of this on paper or you know the just regular um and they are looking out for the company I would say but I think um in general it is back to the instead of working with engineering it's working with the security teams I would say to make sure that um the intent is clear you know what you're doing and then find a middle ground but um definitely there have been several instances where um

sometimes within the security Aug there's a blocker for that we'll do a staggered approach this time so I'm going to go ahead and answer and and Nathan's going to appreciate this but yes uh and Ariel you will appreciate this as well which is uh so there's maybe two examples the first example I'll mention is very current we have a toxic out of SLA vul slack bot okay I'm on every one of those slacks and like the hammer comes down every time but like I there's like a thousand of those every day like that is toxic don't do that and Ariel had a nice talk about don't do that don't be emailing people all the time do other

things uh the second thing is like a cool trick if you use it right which was in the early days uh there were just two of us doing Security reviews and everyone had to do Security review and so to build support for distributed security you know we just kind of maintain that model for a little bit kind of like saying hey this people complain so you just kind of like Let The Suffering build to a certain point and then you give them the next thing and it it only buys you a certain amount of time but if you're already operating in a bad model you can kind of use the pain from your bad model to rationalize

the next model so so for me for this particular answer I usually think about the security engineering relationship and I want to say as an industry we're really bad many years ago where um we always say no and we' shut things down on the engineering side without actually providing them good solution forward but I think over the last decade we've done really well um we're becoming we are engineers and we think like engineers and we provide solutions for engineers so I think overall as an industry we've definitely approached it in a good way and we're really partnering with folks in itself you do have the right political answer so vote for me

um yeah I I I agree I think we've gotten a lot better as an industry but you still do run into people who are kind of like toxically frustrated at everything around them in security anyone no no um and I think in that case like I I've seen that happen in a a few instances and um that's tough because it's usually well-intentioned like I wish that security was better around here um or I wish that people would just listen to me I think is often like the underlying thing um because because you're well-intentioned and because um you you think that there could be a better world and so I I think of that usually as like

maybe can show up toxically but I'm like oh this is me assuming good intent there's there's a good person in there who really is trying to make a difference and so I think then um it does happen and connecting with those folks on like uh ways that they can influence others skills that they can build to bring the data uh to a conversation or um connect with somebody in a way that makes their message more clear because I think when the frustration comes um out ahead then they're not getting that alignment that they need to with their partner teams um and so yes certainly does happen I think there's things that we can um do as

individuals to try and help that right depending on how quickly people answer this this may be our last question but what are some of the SEC security tools that move the needle the largest in being successful at scaling programs I I can start start yeah um I know Eric and Ariel uh rolled out democratized vulnerability management that's my favorite program that I've ever seen in my career um it's made the most impact where in the process of fully utilizing it at ripling right now where um the goal is really making the risk owners actually own their security abilities um so uh in order to do that um what we do is we make sure that

security is not part of the process we're looking at things at the aggregate uh at the high level and holding leaders responsible for their groups of vulnerabilities but in reality um anyone uh can request for SLA extensions I don't want to chase Engineers uh they can fix their own vulnerabilities I I don't want to be responsible for that um so what happens is that our uh P4 p5s um they get SLA extensions if they ask for it automatically approves no one really cares about low information b um right now um and then you go up p3s they get uh they have to be approved by directors p2s have to be approved by VPS p1s have

to be approved by SS and what we've noticed is um in the past VPS were not really aware of the security technical debt that they owned um and once they see an extension come to their plate um they like okay what's happening here why haven't we fixed this um and my favorite scenario of this is when uh P1 came to uh uh P2 came to a VP it came for a second time and he asked the team he's like what's going on here I've already approved this why hasn't been this fixed he's like if anything goes wrong it's my name there and security is going to come after me for this so he asked for the person that's going to fix

it um how long they're going to required to fix it and what the actual plan is before he was going to approve um they did that by the end of the day they said this person this is the plan it's going to take us 6 weeks um he approved it it was done in a week and a half it was done a week and a half because no one was going to come back again for asking for another extension so that's definitely been my favorite program which is why I like I love focusing on that side of things and I feel that it makes a huge huge impact in the business okay I'll go um so maybe

similarly to democratized vul management I think everybody here has taken a democratized approach to uh absec or secure design review probably like security Champions or something else but so I'll just briefly talk about our subtle variant to that which is we hired zero people to embed on any teams and we have zero volunteers so in instead what we did was think real hard about what what we thought the limit of what every developer should do on security which is maybe a little bit less than you might think and then try to frame that as like you're already every developer owns security you already get your code owned by a senior developer who is your peer

reviewer so make it about what the developer can reasonably do and what the senior developer can reasonably peer review and that is kind of the space of what we allow teams to be autonomous with maybe that's kind of vague but the point is is like none of them do security for the other everybody does it for themselves and we make it very lightweight for the partner so that there's no opportunity for it to become a cognitive offloading activity and then what we try to kind of optimize for is when you don't know the answer call us and how to make that very obvious to them do you want to go any want I forgot the question what made you scale best

yeah ah okay thank you I will answer um so uh uh internally we have built um a system that understands a whole lot about all of our applications and so when we go to tell them to do something we have reasonably good idea that what we were telling them to do represents risk to Netflix and means that they should take action we value developer productivity very highly and we consider interrupting them to be um uh something that we will only do if we are pretty certain that they should change Behavior or do something about it um and so by kind of bringing together all of these facts that we can observe about um our applications and things then we can

bring that context with more confidence and also with kind of the thousands of microservices that we have we have to kind of make those trade-offs and we have to um be um well informed about them because uh if we are you know cutting 14,000 Juro tickets we will not be able to answer questions about them at all um and so uh and so bringing all of those facts together um is an internal tool that we've built that I think has really helped a lot I think um I want to maybe like quickly summarize my framework of this in general I think there are in ABS there are things you can solve by uh making sure it's part of the framework and I

think in general programming Frameworks have done this well like you don't think about xss or CF or SQL injection anymore that often um there are things you can only detect by testing or tooling uh to for that and there are things you can't you can do it neither you have to do training um and training is the hardest like scaling training is the hardest I think J spoken about that quite a bit um you can build tooling to detect a lot of your issues that are simple to fix and you should be building those out simple some GP roles will do for the most job you don't need any fancy sast and things in that sense um and then always yeah

just focus on whatever you can build at the platform level I think like uh Julia mentioned we also do have hundreds of microservices if not thousands and so how do you ensure that all of them have the right um configurations and how they deployed and how they are getting patched all of that stuff so building the right developer tools for all of that um I'm conscious in saying developer tools and not security tools because no developer is going to take the time to come log into your security tool and figure out how to how how to work that thing out they're going to like want to find those issues where they are so uh for instance we give all

those findings for a for their team on GitHub uh because that's where our developers are similarly we built the SoCal monocle which youve written quite a bit and spoken quite a bit about which kind of Aggregates all that information to say okay you are a service owner for this service here are the three things you need to do and each of these three things will take you 5 minutes to fix it um yes we could do it but we're not going to do it for hundreds of these services so that's why I really need you to come and do it for you great well we only have two minutes left um don't think we can answer another

question okay so thank you everyone so much for being engaged and thank you to our panelist for a great discussion on scaling [Applause]