Building Sustainable Security Programs

enough rambling for me um so i want to introduce the real content now our first keynote today is building sustainable security programs asta singhal is currently a director of security at netflix leading teams responsible for securing netflix's workforce and product technology footprint in support of the product studio and enterprise prior to this she was a product security leader leading security for the salesforce app exchange and other core products she is passionate about proactive security by design scalable security programs and inclusion in the security community without further ado welcome to stage [Applause] [Music]

hasta right very good morning everyone my name is asta as reid said i lead workforce and infrastructure security at netflix and besides is one of my favorite conferences in the world so i'm really really excited to be up here today and today we are going to talk about building sustainable security programs okay you may be tired of talking about information security burnout because you know it's a topic that we've all been talking about a lot maybe more so than we ever have before especially after two years of the pandemic and counting thank you everyone for wearing your masks by the way um and in the pandemic when it's been difficult to just be a human let alone be a human

in a job where you just never win and of course there is no shortage of literature on burnout in the infosec industry and all the underlying reasons that go into it but um today the topic i want to talk to all of you about is the ways in which factors that contribute to the pressure and stress that we feel as security professionals and changes that security leaders can be making to their organizational culture to their risk perspective program strategy and alignment with their stakeholders that would help improve the sustainability and well-being of their teams okay so why don't we jump right in we will start from talking about the contributing factors that play into this

often in infosec we're in firefighting mode a lot there's always another open vulnerability that we have to go chase down another feature throughout model that needs to get done another sas security review another java vulnerability that has to be patched across your whole fleet we can just never catch up and being in this heightened state of stress and constantly reacting to things it's exhausting and when you're in this mode yes you may be keeping busy all the time but you aren't always being effective at addressing the root cause or even making the highest quality decisions all the time and as we all know security people we care a lot in fact sometimes we care a little bit

too much and if you have spent any amount of time in infosec you've been disappointed you've been disappointed by the decisions that were made by your stakeholders you've been disappointed by the state of things the bugs that got shipped to production the intense back and forth with a product team that still led to a risk exception and because of all of that disappointment we sometimes tend to give up a little bit we tend to assume the worst of our partners of our customer teams of our stakeholders of our leadership it's almost like we've earned our cynicism we think oh i know how bad this thing is if everybody else cared as much as i do

but of course that never happens so it's so easy to become jaded by the state of things i definitely remember a number of times walking away from stakeholder meetings and just feeling annoyed and when you're in the business of preventing bad outcomes and you're constantly surrounded by all of the things that can be going wrong and there is a high impact of that thing going wrong then it becomes easy to catastrophize everything we get so focused on the impact of what's possible that we forget to consider what's probable and it's easy to assume that the worst case scenario it's bound to happen it's going to happen this year it's going to happen this week this month

there's no denying it we're just waiting for the other shoe to drop and our community is full of really passionate people that truly enjoy this field and information security we really care about the work and we put a lot of ourselves into that work and for most of us this translates into feeling like our job is to prevent every bad thing from happening and there is enormous personal responsibility that we all carry around with that and because there's a sense of personal failure there and when you couple that with the fact that your customer teams or your leadership don't really understand what to expect from their security program that is what they think right we have a

security team so nothing bad can ever happen i want you to take a second and think about how ridiculous and impossible this expectation is that we've just made our normal and then we just walk around and live our life and we're like yeah that's the i'm ready to carry around that pressure every single day

and more often than not there is tension and conflict between the security team and your stakeholders you're always convincing someone to fix that bug to prioritize deprecating that one legacy service that has all those security holes or you're escalating risk exceptions in product features that are going out the door or you're denying that sas vendor request that the marketing team has to use and of course there is supposed to be some healthy tension there but being in this state of constant conflict with your customers and stakeholders it weighs on you it's hard and the changing threat landscape doesn't help either there are new emerging threats all the time that may impact your posture what was good enough yesterday may not

be so today maybe you weren't thinking so much about ransomware a year ago or software supply chain issues a couple of years ago but now it's all you can talk about and the vendors won't stop calling you about it and now you have to go back to the drawing board and figure out all of the assumptions in your program that you have to now go revisit and after all of that no matter how hard we try we're never really done it's not like one fine day we'll all wake up and say okay we're all good we've secured all the things forever we can all just go home no we can never make progress fast enough

and there are never enough things in the wins column there are always more broken things to fix in fact there isn't that much we can point to as oh i did the thing that's the win right only one thing has to go wrong to result in a bad outcome

okay so i just want to acknowledge that that was a lot and that is the reality of our community and our profession today i have personally struggled with a lot of these things and i know folks in my community that have in fact when i was doing a dry run of this talk with my team they're like this is too depressing i was like this is our lives but the good news is that we do have the ability to change this status quo so why don't we talk about that the first thing that i really want us to focus on is how the culture of our security organization impacts the sustainability of our teams we talked about how easy it can be to

become jaded and be cynical about the state of things but the problem with this individual cynicism is that before you know it that becomes the culture of your team and that can be really really harmful in your ability to show up for your customers because you can't be objective about the perspective that they're bringing to the table you don't trust their intentions you don't trust their judgment and you can't show up as a collaborative partner because you assume the worst so as leaders it is our job to actively disrupt this kind of mindset on our teams so the next time someone on your team is frustrated because some team is not fixing a security issue

help them contextualize the security work in the broader context of the business goals it's entirely possible that the security work is not the most important thing that needs to happen for a particular launch guide them to find the ways in which there is commonality between the security team's goals and the business goals help them to understand the customer team's intent and work towards a compromise that may still mitigate the risk that you're worried about and the other way in which we as leaders impact the culture of our organizations is through hiring so when it comes to hiring for your teams you need to be thinking about skills like customer enablement collaboration stakeholder management as much as you think about technical

competencies for a particular role maybe more so because these skills are really really important in working collaboratively to solve problems with our stakeholders and to build a healthy team culture and this jaded mindset of like oh man everything's broken or developers they just don't care about security that should be a red flag we need to be paying attention to these types of things when we are building out our organizations i came across this comic in a blog post that ryan nakomoto wrote about heroic's culture now of course the idea of the person that saves the day it's really appealing who doesn't want to be a superhero i would much prefer to be wonder woman but the

fact that you need heroics to save the day means that you failed earlier on somewhere and the problem with the heroics is that it's not sustainable it's exhausting if you have to put on that cape constantly i'm not saying that incident response and firefighting is not important but what's also important is making the proactive long-term investments that would reduce the need for as much incident response it would focus on scalability sustainability and it's no surprise that you incentivize more of the behavior that you reward on your teams and of course when somebody goes above and beyond you want to recognize their efforts the hard work and the impact that they're having but we also need to get better at

digging deeper and understanding why that is heroics going on on your teams can you understand the root causes that are at play here that make the heroics necessary and can you solve for that instead as an example if you run any java in your ecosystem you probably remember your log4j response from a few months ago i know i do um and my team you know for that response we made stickers then swag for everyone that participated because you know we like to do stickers and swag but it was great it was good to recognize all of the hard work that went into that but at the same time when we were conducting our retros we also focused on the success of the

long-term investments that made that response possible and efficient so things like ecosystem visibility things like our well management tooling all of these things are important for us to be able to do incident response efficiently and as part of those retros we also try to identify areas where there may be a single point of failure or areas where we needed more run books or better ownership because those are the things that will enable our long-term success so when you conduct incident retros make room for understanding the factors that impact the sustainability recognize the painstaking long-term work that creates an impact celebrate the work that prevents the need for heroics all together culture it takes intentionality it

doesn't just happen so setting this tone for sustainability and long-term thinking will foster a culture on your team that values it so this idea of additive teams really resonates for me and i wanted to share it with you all it focuses on the need to assess your current strengths and gaps as a team to determine what you need to be hiring for so you can add to your existing perspectives and skill sets so think about the competencies that your team currently has the feedback that you receive from your partners and customers the opportunities that you aren't quite able to realize yet and map that into the needs for the next person that you need to hire on your

team maybe you have a lot of technical security competencies on your team but you aren't able to improve the product experience of the ex the products that you're building for your customers or maybe you're missing program management skills to take things to 100 and actually realize the risk reduction impact or maybe you have folks who can help assess the severity of a vulnerability but maybe you're missing the engineering chops for being able to do things like automated pull requests that would make the remediation process even easier for this introspection about your strengths and gaps can really help build team trust increase self-awareness and help folks be more thoughtful about okay what are we missing and really help them

appreciate the skills that they need on their teams as an example at one point my team was largely absec experts and they were all really great at what they did you know finding vulnerabilities security architecture all of that good stuff but we were not going to scale our program with just abstract focused work so we grew our team to bring product focused security ics to bring program managers software engineers infrastructure security engineers because we needed all of those things to achieve our long-term mission of being able to build a scalable application security program and thinking about team composition in this way can create an environment in which folks can bring that new perspective to the table

and they can be really deeply energized by the outcomes they're able to achieve with each other and what they can learn from each other and it also reduces the commiseration and pessimism of the battle scars that all of your security people may share and trust me i think your security people could use the optimism and it gives you the opportunity to bring new ideas that can help solve some of these age-old problems that we may have given up on a little bit and we need new ways to think about them

when it comes to building an empathetic team culture team leaders are the ones who really need to take initiative you set the tone for what that looks like on your team you sit in a position of privilege where showing up with vulnerability may be easier for you than most folks on your team but this vulnerability can be really powerful for your team culture and it can lead to really meaningful connections this job as we all know it can sometimes take an emotional toll on you and having an environment of vulnerability and honesty can really help folks feel less alone it allows folks to find community with their peers and it's beyond simply being able to

collaborate on the work that they're doing it's about being able to build connections have the support structures in place that would allow you to navigate difficult topics and relationships with customers and even help each other recognize and celebrate the wins a few months ago one of my teams decided to spin up a forum to really discuss how they work because they wanted to learn from the techniques that each of them are applying to kind of like their own area of work and sure they got to do that in that forum but it also ended up being a really supportive space for them to share their challenges and to support each other because they had built that trust and

that environment of empathy and vulnerability and they were able to feel seen and heard with their peers and do their best work and such environments can also help us get through those difficult days when you just can't seem to win the criticality of the security work that you do it may vary based on the business that you're in for example human safety will always be more important than say credit card information but no matter what your business is one thing that's mostly true is that security teams exist to enable the business now of course our job is to manage the risk for the company but at the end of the day if the company was to go out of business then we

wouldn't need to be here it's really important for security teams to internalize this mindset of business enablement because this helps us think of ourselves as a department that's invested in the success of the stakeholders that we're supporting we're not here to do our own security thing in a corner and we are here to be a part of the success of the business that we support and it doesn't have to be so isolating in fact i would urge you to go one step further and think about what would it mean for you to be a customer-focused security team can you provide services and products that your stakeholders are truly delighted by and i'm not saying that we have to stop

advocating for the right things for security and then we just give up all our influence and we just do what the product managers say no that's not what i'm saying what i'm saying is we really think about those security outcomes with influence and building a track record of a security team that people truly enjoy working with in that world i think it also makes you think more about what are the security experiences that aren't as usable that people are trying to avoid and you can focus your energy on improving that for example if there is a cumbersome manual review process that folks are less than thrilled to go through what can you do to reduce the number of

steps in it can you automate more of it can you make it more secure by default think about ways in which you can build customer focus into your security team culture the second area of focus that i want to talk about is building a focused perspective now hopefully we can all agree that a security team's job is to manage and reduce risk so what is risk risk is likelihood times impact the total amount of risk exposure is the probability of a bad event occurring multiplied by the potential impact of that event but i actually think security engineers were really bad at risk we get hyper focused on impact and we forget to consider likelihood which is why everything is always doom

and gloom and to be better at our jobs of managing risk we need to build a better perspective on how we think about risk risk impact so thinking about probability and not just the impact of an event ryan mcgeehan who some of you may know as magu has done some great work on helping engineers understand and speak risk i highly recommend you check it out i will leave the link in my slide that will be shared afterwards and this idea of risk forecasting and quantification it can really help us think about it in more concrete and time-boxed ways and it helps us move away from preventing all bad outcomes to reducing the probability of the bad

outcomes it helps security engineers be more realistic about risk and the ways that we can mitigate it now i want to acknowledge that this is hard and it's imperfect but it's imperfect in the same ways that what we are doing now is imperfect so i think it's worth the effort to evolve how we think about risk security teams we also need to build a clear understanding of the prep model that our business is operating in and this should actively guide how you invest in your program let's be honest not all companies out there are protecting against nation state actors so you need to know what are your critical assets who are your threat actors and what are their

motivations this can help you be more measured and deliberate with your security program and understanding why security matters to the business can really help contextualize your program in the broader picture

now based on the threat model that you operate in your security team may have a certain level of authority to be able to say no to things or maybe you don't have very much authority at all you're expected to operate with influence and advocacy no matter your situation i would strongly urge you to operate with the principle of holding yourselves accountable to a high standard on what is the security guidance that you're providing and making sure that it aligns with the risk perspective of your organization if someone was to show up at your door tomorrow and ask you explain to me how this decision was made what is the impact on the company what what is the assumptions on which

we're operating we should be able to be transparent about that so we need to put a lot of rigor into the security decisions that we make for our stakeholders be more pragmatic and transparent with your guidance this will not only help your security teams be more thoughtful and pragmatic but it'll also improve the experience of your customers

okay so hopefully we can all agree that we are never going to do all the things but there is a lot of things that need to get done every day so having a strategic program focus is really important for sustainability and scalability of your program and your teams now there's obviously a lot of important operational services that a security program provides to the company individual threat models feature reviews incident response wall management all of that fun stuff but sometimes it can be really easy to get lost in that operations and lose focus of the strategic investments that would enable your long-term success so once you have a clear idea of the needs of your organization

it's really important to find the right balance between spending your time supporting that day-to-day operational work versus investing in that long-term strategic work that lets you get out of the day-to-day so if your abstech team is spending most of their time doing security reviews they may not have the bandwidth to really think about oh is there a way for me to make certain baseline controls just default across the board but you can't make those investments if you're always just in the operations and don't have the bandwidth to look up from the day-to-day you can't patch the boat if you're always just bailing water as security leaders we have to get better at recognizing these patterns on

our teams and make room for them to invest the time in the long-term strategic work that would get us out of the day-to-day operations or streamline it over time in addition to this we also need to get better at clearly articulating what are the overall goals of our program and what should our teams be doing to get there for example is your goal to prevent most bugs from getting to production is your goal to get everything fixed within a certain sla is your goal to just have baseline security guarantees and good observability across your fleet it's important to be clear about those focus areas to your teams because this would allow them to make day-to-day

decisions that are really in line with the assumptions of kind of what are you here trying to achieve the answer can be we're here to secure all the things because that's just not reasonable it's never going to happen because we can never do all the things and i think that's true no matter how well-funded your security program is i'm sure you go to you know the largest security team in the bay area and they will still tell you that they don't have enough funding and it's important to set those strategic goals that are in line with your threat model and you know how much investment you're able to make in your security program so that your teams can

make those thoughtful and measured decisions every day one tool that we use for this at netflix that i really like is this idea of strategy bets so strategy bets are basically informed judgment calls on what what are the bets we're making about our program these things need to be tested debated and altered over time so it's not a 100 obvious choice but it's one of the two possible routes that we can take and this idea of taking one of those possible routes it allows us to be deliberate about not choosing that other route and investing our energy into that so for example our security team has chosen to focus on this idea of leaning into the pave roads and securing those

central points of leverage now the other option there is to go meet the customer teams where they are and secure every possible technology option i think that is also a valid path but we have chosen that not to be the one that we are investing our time in and knowing this everyone in the organization is able to be really deliberate about the investments that they make day to day and be confident that that's aligned with the overall org strategy

another area on program strategy to think about is where can you find leverage points and efficiency in your program and sometimes that may even be outside the security team some of you may have heard me previously talk about this idea of hitching the security wagon to developer productivity which talks about finding ways to scale security through developer tools and experiences some of my colleagues in information security and cloud gateway published a blog post last year on how we did this to productize certain security controls through our authentication proxy so in that case we approached our security checklist with sort of the mindset of okay what are the things that are consistent enough across all of

these applications that we can easily build into that authentication proxy so things like the web application firewall ddos prevention security header validation consistent logging all of those things fit the bill and what ended up happening is that this improved sort of the internet facing controls work not just for our team but for every customer team that wanted to put a new app on the internet we're now trying to realize similar scalability wins with other parts of our infrastructure as well and this not only helps scale our program but it also makes security a true partner to other central teams at the company that we get to work with think about how you can create the time

and space for such leverage focused work on your teams can you make this specifically a part of your team's charter can this be someone's full-time job because these types of leverage investments are important in that long-term scalability and sustainability and in the world where there are a million things to do thinking about critical data assets that have a real impact on your business can be a strong guiding light on how to invest your time so instead of closing every possible door and window could you lock down access to those critical data assets and invest your time in preventing and detecting potential impact to that can you find ways to minimize the exposure to those critical data assets

figure out tiering criteria to be able to right-size your investment based on data categories top risks should always be the context in which you make your investment decision both for proactive and reactive security controls and speaking of proactive and reactive security controls i think there is a reframing here on overall security assurance for your program as opposed to preventative versus reactive work right so at the end of the day there is an opportunity for us to really think about how do those investments complement each other let's say you can't roll out comprehensive authorization policies for a new low trust user type in your ecosystem for another three quarters could you work with your detection team

to put some detections in place that would monitor anomalous activity from that particular identity type would that help your security engineers sleep better at night in fact you may have areas where your ability to put in proactive controls is just limited altogether let's say you can't fully lock down your ci cd ecosystem because let's be honest all your engineers will hate you and it'll be a developer experience that is pretty hellish to work in so what do you do can you put locking controls in place to detect potential attack scenarios that are fairly common like attempted packet squats and other ci cd type issues in this way you can utilize those reactive controls to compensate for the

limited proactive controls work while being able to balance the overall security experience of that particular world and last but not the least is the investments that we should be making in stakeholder and leadership alignment more often than not senior leaders do not know what to expect from their security program and of course we talk about how well it's not their job to know it's our job but the problem with that approach where they're not supposed to know what to expect guess what what they expect is nothing bad can ever happen because we have a security team and then when it does we'll just fire the cso and move on with our lives and that leads to all of this

dysfunction of you know living up to that impossible expectation of nothing bad can ever happen so as security leaders for the sake of our teams we need to have a meaningful conversation about risk appetite what are the top risks and what does level of investment in the program mean for your ability to mitigate those risks versus not now we talked a little bit about quantitative risk before that can be a great tool to make this a dollar to dollar comparison about risk tolerance and security investment and it will allow you to create clarity on what's important and how can you further improve that story with further investment now i'm not going to try to explain

quantified risk to you because i'm not an expert in this space and i'm lucky enough to work with people who are experts in this space um i will be linking out their work on risk quantification and a blog post that they wrote about a risk quantification library that we open sourced a few years ago so the link will be in my slides for that and in this world hopefully you're also able to work towards some shared guiding principles between yourself and your engineering and business counterparts so let's say if the guiding principle is that you need to prioritize innovation above all else then maybe the business is able to lean into more security risk and incur the security cost that may

come with that or if the guiding principle is that you want to aim for a high prevention standard on the security controls then you can align with your business counterparts and your engineering leadership on what that will mean for velocity or developer experience because let's be honest we're not going to get all the things all at once you have to make some trade-offs somewhere and having these shared principles with our counterparts allows us to be thoughtful and aligned on those trade-offs that we need to make and there can still be a healthy tension with those stakeholders in this world but at least you won't be disappointed by their decisions as much because you're making them against agreed-upon

principles or you won't be in this constant state of conflict and relitigating everything with them all the time now with this idea of having shared principles clarity on top risks risk appetite seems like we're going to a good place it's also important to create ongoing visibility for the top risks to your stakeholders and to senior leadership all security teams more or less know the stuff that keeps them up at night and more often than not it ties back to the top risk areas for your company and sometimes this can also be the stuff that that one security engineer who feels a really really strong sense of personal responsibility carries around with them every day oh that one team they have that legacy

app well hopefully nothing bad happens with it and when you create this visibility and shared ownership then the accountability exists along with your stakeholders the security team's not carrying that around by themselves they're not worrying about it alone in a corner and when you have those types of forums you can utilize them to highlight the shared wins of the program and the ways in which you're able to make progress and improve those relationships as well and lastly i cannot overstress the importance for showing up for our customers with reasonable expectations that are in line with your risk appetite in your guiding principles for the program don't be the dentist where no matter how hard you try you always get in trouble

it's like oh you brush twice a day how come you don't floss no no nobody wants that we need to show up for our customers with the reasonable approaches that align with your risk posture okay so we covered a lot of ground there why don't we do a quick recap for our key takeaways there are a lot of factors that can go into the lack of sustainability for security teams ranging from personal cynicism to sense of personal failure to evolving landscape and stakeholder conflict we as security leaders can help improve the sustainability of our teams and our programs by focusing in a few key areas first and foremost building a sustainable empathetic organizational culture in which people

feel supported energized and excited by their work is really important you have to disrupt the cynicism you have to be intentional about team culture you have to celebrate the wins and you have to solve for long-term sustainability it can help build a supportive team environment and make security a team sport second you have to invest in building a thoughtful pragmatic risk perspective that helps your security team understand why security matters to the business and how they can show up as thoughtful partners to the business this can help combat the ongoing conflict that we feel with our with our stakeholders make security a part of overall business success and help us get out of that doom

and gloom mindset third is you have to be very deliberate about the strategic strategic focus areas of your program we will never do all the things we're not superheroes so as security leaders it is our responsibility to set up our team in such a way that they have the bandwidth for the strategic long-term investments there is a clear understanding of the focus areas and the guiding principles of your program that can guide day-to-day decisions that they're making and help with measurable reasonable approaches stronger cross-functional relationships and allowing the team to focus on what's really most important and last but not the least you have to build strong alignment with your stakeholders and your leadership so you

have a shared understanding of risk appetite and the guiding principles that inform your organization it can help ease the pressure that your security team feels day-to-day to relitigate expectations and carrying personal responsibility for areas where there may be some reasonable risk acceptance if we invest in some of these things we can build sustainable and scalable security programs that's all i got thank you so much [Applause] thank you asa for presenting it beside san francisco on behalf of the conference and our sponsor gif uh our gift speaker sponsor maltigo we want to present this gift of our appreciation thank you again

is there time for a question sir um yeah if there's any um you've got about three

you minutes your talk in in terms of security and external that how in your experience which one is toughest like how is it how much time do you spend on the internal stuff for your team versus the i think as senior leaders oh sorry i'll repeat the question so how much time do you need to be spending internally with your teams versus externally with your stakeholders and your leadership i would say for senior leaders of the security organization your primary job is that external alignment with leadership and stakeholders that allows your teams to execute well against shared principles so yeah i think security leaders need to think about how they're spending their time and it needs to be more outwards

all right thank you very much