A Blameless Retro on Security

[Music] so uh I want to start off this morning before I get into my content with a bit of an acknowledgement a land acknowledgement this is the Portland State University land acknowledgement and I do want to apologize in advance if I mispronounce any of these tribes names but we honor the indigenous people where traditional and ancestral homelands we whose traditional and ancestral homelands we stand on the mola Cathlamet Clackamus tumat Wala lands of the chinuk the tulaan kalapuya and many other indigenous nations of the Columbia River it's important to acknowledge the ancestors of this place and to recognize we are here because of the sacrifices forced upon them in Remembering these communities we honor their legacy their lives and their

descendants so preparing for this talk um was hard because uh security kind of sucks right now everything's not going well just this week I got contacted about the Caesars uh breach because I have yet another two years of uh identity monitoring I think at this point my identity will be monitored long past my breathing uh we've got uh Clorox announcing the financial impact of their bleach breach and uh MGM talking about $100 million in losses and it's not like we don't know where the attacks are coming from or how they're being carried out um nobody wants to talk at a big conference about fishing fishing's been going on for years at the basic con we all want to

talk about the exciting uh new oday the elaborate attack uh the the new way that I you know popped Cal on on Windows or something and the reality is you don't need oday um and so uh security is kind of rough right now and it's hard to be optimistic and uh especially when all I want to do at this point is uh go raise baby goats Off the Grid and mutter about technology um so so that's not what Keynotes are supposed to be Keynotes are supposed to be inspirational we're supposed to be optimistic we're supposed to get People Pumped up for the day we're going to learn about security it's going to be great so um let's dig deep and find that

inspiration in all the things we're doing wrong and it's really easy for us to talk about oh if the developers would just if the users would just and I'm here today to talk about what security is doing wrong and how we need to change if we want to get different [Music] outcomes so first off if security were just a technical problem we would have fixed it by now uh how long have we known about input sanitization like we know how to prevent those vulnerabilities but we keep releasing code that includes them and that's because security is far far more than a technical problem people write code human beings and they're they're writing code to do a job and and a lot of developers

will tell you they got into coding uh because they're inherently lazy they didn't want to do a thing manually and they wanted to automate it and they wanted to automate it quickly and that is not always conducive to high quality robust code they're they're getting a job done and they're gold on getting a job done and even if they are quality conscious humans are notoriously bad at assessing risk and uh you know after something bad has happened you know their Shields go up and they they're a little more conservative about things and they're more aware of the risks uh presented to them but after a while they become complacent again and so we keep applying

Technical Solutions to human problems and glorifying those those tools that we can apply and undermining or or discounting the the human factors defs Ops is popular because security teams aren't developers like the idea of Dev SEC Ops they love the idea of shift left let's stop throwing a bunch of resources at the security team you're going to give me SAS tools and software component analysis tools and I'm going to get I'm going to get all these tools great and then I don't have to deal with the security team because they're always showing up telling me I can't do a thing or that I did it wrong or I have to go do this other thing that

isn't what I'm gold on that our customers are asking for and so from an executive and Engineering perspective defs Ops is great the problem is we're throwing tools over the wall to Dev Vel opers who don't have security training this this idea that um developers will do the right thing if they have the information that that assumes that they know what the right thing is and and all of our all of our tooling is not helping them get any better and here's how devs Ops is doing it wrong we've created dashboards of trash that developers have to go look at that are completely disjointed that aren't anywhere in their workflow and we expect them to go go look at the sneak

dashboard go look at the sonar Cube dashboard go look at the no they're not going to go do that and then what manually create their own jira tickets no they're they're not looking at

them these tools are reporting every sometimes they're reporting you're using a component that has a vulnerability that is a CVSs 2.9 okay is that really important for me to go fix right now do I need to be interrupted with that what do you mean I have 800 new vulnerabilities this week how many how many of them are critical oh okay 45 of them are critical so I have 45 bugs to patch wait a minute the first six I looked at were all in the same library and they all told me to do the same thing why did you cut me 45 bugs that have eight actions Associated as a developer it's a lot of noise for them

to sort through and then you get to the bugs where it's like you have this bug and uh no I actually don't because that's just in the wrapper for the package or yes I use that library but we're not using the vulnerable component like the actual vulnerability isn't in our application it's just that a pattern match was found that yes I use the component but I don't use the vulnerable code so we're we're drowning them in noise

uh I think these are all pretty much realities that we experience every day in terms of we're pushing things to developers who don't have the training and don't have the uh ability to discern do I need to drop everything and fix this right now it's just an interruption it's that buzzing fly around their head of uh more stuff and uh that's part of why they don't go look at those dashboards because it's a distraction we heard yesterday uh if you were in the uh GitHub talk yesterday afternoon that developers one of their big complaints is interrupts in what they're trying to get done and the security team is nothing but an interrupt showing up and trying to pull

them out of what they're working on no no Dev team I have ever met plans just an open block of time every Sprint for security maintenance we're always interrupting them and they're always having to make tradeoffs on the business plan to fix these security issues and the security teams are frequently underst staffed to meet the needs of the dev teams so the developers are like you're giving me all this crap which one's really important and the security team is saying I can get to you in a week and a half well I'm out of SLA by then so that doesn't work for me I'm just going to mark it as by Design and close it um

which which also happens and so we're creating this tension which is not getting the right thing done and so what's our solution to this more security Engineers but we already talked about security is not always a technical problem sometimes it's documentation sometimes it's training and you know what Engineers really love doing writing documentation no not only do they not love doing it they're bad at it and you're paying them way too too much why would you spend an engine anyway I like I said this talk turns into a rant about all the things we're doing wrong we keep hiring security engineers and applying them to non-engineering problems so we're back to where we started which is I'm going off the grid

and I'm going to raise baby [Music] [Applause] goats okay no let's just try something new first off let's admit that security Engineers aren't good at everything let's just own it and say you know what a security engineer who's really really great at threat modeling might be terrible at incident response and a security engineer that's really good at vulnerability reproduction is not the person you want interacting with the bug Bounty researcher in many cases um who's going to write the customer documentation who's going to do the the business analysis um of risk Trends who's going to go train the developers um on how to do do better coding um at the end of the day I keep

seeing companies hiring more and more security engineers and making them do project management making them uh you know do Project work engaging stakeholders communicating timelines and it's like you know you could just hire a security program manager and let your engineers do engineering work um that is what you hired them for and so uh we need to start accepting that uh you know there are sometimes Partners who are better at doing things than we are as Engineers this one usually gets a few people in the room uncomfortable and that's okay but the way we've implemented security champions and our Dev teams is fundamentally flawed it is bad we it's just like throwing a sass or a Das tool at the dev

team and saying congratulations you've shifted left um we've put these security Champions out in the product teams and in the engineering teams with no support with no structure with no managed intentional community and what happens to them they don't report into the security team security is like their side hustle project um it's probably not even on their performance review they're not gold on it they report to a leadership team that wants to ship product that wants to ship a feature that wants you to sign off on this feature and what do you mean you don't think we have a secure architecture we have left them unsupported and they have no community and they don't necessarily

know the security champions in the other teams across your company and they're not sharing tools because they're all disconnected and I cannot tell you the number of times a security Champion has said to me I need to leave my team and go work in a security organization if I actually want to do security because there is no career path for me here as a security engineer in a product team and so uh we need to do more to support our security Champions and that has to come from the security organization um there's there's a field in security considered uh Community Management uh working with the security researchers and Industry Partners outside the company but we need that

internally to our companies to help Foster and cultivate that relationship because security Champions could be your diplomatic ambassador to the product team they should be the person who's working with you so closely that you know that they've got a major Milestone coming up so you don't get surprised a week before release when they show up and are like Hey we're releasing next week I'm sorry what uh have you have you done any threat modeling um what what is your response plan like what is your patching Cadence uh have you have you done any of the security requirements that we've documented and so they can be a tremendous asset but only if we nurture that Community atlassian has a

dedicated program manager who their job is to cultivate that internal community and make sure that they are all connected with one another and that they have meaningful conversations and content and training available to them and that they're getting recognition for doing great security work within the product team and so it it does but it does take that dedicated program manager who that's their job and we can't just assume that these things are going to organically happen on their own if we don't make it intentional I would like everyone in the room to read this slide out loud I'm not even kidding stop rolling your eyes at compl liance yes there are bad compliance Auditors yes they ask for ridiculous

things sometimes I had a job where my team complained about the uh poam report and the uh compliance requirements to do vul scanning and I said hold on are you saying that we shouldn't have to do vul scanning like no no no we should have to do V scanning I'm like do you understand how bad the state of security is that an organization had to make it a compliance requirement because companies weren't doing it compliance is the only way your customers know that you're doing the right things for security how are they going to know that you actually do um you know secret scanning how are they actually going to know that you've got any of the basic things that you should

be doing compliance is basically Consumer Reports so what you'll end up seeing is that security conscious customers are looking for your compliance certifications not because they care not because they're a they don't care that you have fed ramp because they're a Federal customer they care that you have fed ramp because they know that means you met a certain bar of quality now as I said at the beginning there are there are Auditors who ask the wrong questions um who are are overly pedantic about what the words in the control say versus yes I achieved that by doing this um and so that is challenging I'm not going to lie but if you're working in partnership with your compliance team

from the outset compliance is just the scorecard that your security program is effective and so building a healthy partnership there um and making sure that uh there's alignment can really change the game from we're doing security because compliance makes us do it to compliance is simply measuring that we're doing the right things um and ultimately that's that's what we all want is to do the right things and then we get the cookie at the end uh whereas in most organizations compliance is the stick that you get beaten with to do the thing you have to do it because compliance um that's one of the reasons honestly product teams hate security is because we show up with sticks all the

time and never carrots uh it's never congratulations you fixed all your bugs you guys are awesome we're going to like buy you lunch it's you have to fix it because of compliance and you're going to get a bad scorecard report that goes to your VP um we're always negative and we're always uh punishment oriented versus um rewards and intrinsic value and compliance is the biggest stick of them all um and so if we need to redefine our relationship there but we also need to redefine our perspective on security security is not actually special we just QA and I had a big big argument the fact that you're clapping is awesome uh 10 years ago I argued with

a friend of mine uh who told me H vulnerabilities they're just bugs it's just QA and I was so adamant that no it's different no it's it's actually not he was right he was absolutely right um and what's interesting is uh if you start if you stop talking to your development Partners about vulnerabilities and start talking about security bugs it shifts the perspective in their mind vulnerabilities are the security team's problem bugs they own they understand it and if you redefine your security program within the confines of their existing quality program go reach out to your product Architects and say hey I would love to integrate Security in your pre-existing quality mechanisms and not just layer it on top

so we've got security standards or requirements or whatever your company calls them why are they separate from quality because quality code is reliable quality code is um well documented quality code is accessible quality code is secure and no developer is like yeah I write lowquality code uh I mean they're not excited about that they want to write high quality code and so shift it go work with your architect partners and get integrated in what they're already doing versus treating security as some special unique thing that has to be different all the time in my last role we worked with the product Architects to integrate threat modeling in their existing design change process so if you had a major design

change or a new design you had to go through the design dock like process and you had to submit an architecture and a data flow diagram and a hey wait data flow diagram that is something we need for threat modeling anyway so we worked with the Architects to integrate processes around hey we we developed a 30 minute threat modeling training specifically for engineers it's 30 minutes it's really simple it's it's very clear and it broke it down into four really simple questions that we could use every day in our lives and you do this inherently without any specialized training when you think about what are the risks in my home and depending on where I live and

depending on what's in my home I might have different risk tolerances than the person three seats down from you today who lives in a different neighborhood with different Assets in their home and and so they're looking at different uh different risks and how they address those well we we took that realworld uh explanation of threat modeling to the engineers and said just keep asking yourself questions of what could possibly go wrong what could possibly go wrong okay well I mean this is unlikely but it could possibly go wrong and just like until you feel like you've exhausted all of the what could possibly go wrongs and congratulations you've just done a threat model and if

you do that in your design phase you're going to design a higher quality product and so we put threat modeling not as a security review function but as a quality design function and that changed everything for the engineers and their ownership of it and so I just come back to we have to change how we view our role in software development um from the security cop at the end of the process preventing you from releasing or coming back and telling you all the things you have to fix to part of the quality organization and let's be honest when companies started doing away with dedicated testers and being like yeah the developers will test their code a

lot of them set up bug bounties and called it good they're like all right we've still outsourced the testing because developers aren't always great at testing this is their baby and they have blinders onto all the things that could possibly go wrong and so we've just taken the place of the QA organization and a lot of companies and they can't stop funding security C Clorox MGM like they they can't stop doing security like they can shift left all of that test work but we can redefine how we engage with them to get better outcomes so when I say know your customers I don't mean know your development Engineers I mean who buys your product who gives your company

money for a thing for a service for a piece of software for what's important to them because if you understand what's important to your customers you will have an easier time talking with your product teams about hey I'm not just saying we need to do this security thing because the security team thinks it's important we understand what customers need and what's important animportant to them if your customers are saying hey logging into your application is really hard I'm an mssp multiple customers of mine use your platform and it takes me 12 steps to log out of one customer and log into a different customer because of your authentication flow the security team might automatically be like security

that's how it is like we have to be look there's a way you can design a better authentication flow and still be secure but you have to understand the real pain points of that customer to see why this is a problem and so take the time to talk to some of your customers talk to the customer support and sales reps to understand what are they hearing from customers and don't just sit in the security Ivory Tower and uh make rules about what can and can't be done without understanding the user experience and and what the implications of that are because when we we design highly secure poor user experience Solutions users find workarounds and then all of our Security

Solutions are for not because they're avoiding them they're finding ways around them or they're going to a different product that has a better ux so at the end of the day you need to actually understand what your customers need and want so that you can prioritize the right things in security we also need to stop letting perfect be the enemy of good I keep seeing security teams do this across the industry and it's and it's not only damaging that team but uh the the engineering groups that work with them so we will come in and ask for the perfect solution which is like asking the dev team to speedrun Mount Everest and they can't do it and and your

expectations are so wildly unrealistic for them they don't even try but if you look at what they're doing as a as a product as a engineering solution as an IT team and you're like wow this is terrible um and we've all had that where we come in and we see what somebody is doing in their engineering solution and we're like oh okay uh where do we start uh the reality is you have to be pragmatic and you have to help them on that Journey you have to get them to base camp one and let them equilibrate to the altitude there and then you rally them to base camp to yes it is important that you and

your leadership team know in your mind we got to make the summit we got to get there eventually because where we are now is unacceptable but we have to do it iteratively and and we have to get there a little bit at a time I had an employee who was uh very frustrated because uh they worked really hard on securing the devops pipeline and the cicd pipeline at a company and uh we got about I'd say 60% of the repositories into a better condition with Branch control because not everything at the company was running through the same GitHub pipeline some was GitHub Enterprise some was uh GitHub org because it was open source and and

we only got Branch protection on part of the environment and they felt that they had failed and I was like you realize we had Branch protection on nothing before so we're better now like we have iteratively gotten substantially better than we were 3 months ago why are you beating yourself up as a failure this has been in my mind a wild success and now that we've got it done in one environment and we have it as a case study where we can show this didn't add substantial development time in the process we did reduce the number of bugs we did increase quality now we have the data to push for it across the entire Enterprise so we have to be willing to

accept iterative improvements and uh make those trade-offs and one of the things that's really challenging here is security people are used to getting in trouble for everything that goes wrong why didn't you catch this why didn't you stop this why didn't you prevent this and that I think makes us afraid to accept good we're always looking for perfect because what if we're wrong what if something goes wrong and that's where it comes back to yeah you want to you want to document the risks and make sure your leadership team knows like here's all the things that I think we really need to do but realistically this is what we're going to get done this fiscal

year and maybe this first quarter all we're doing is socializing change we're not even changing anything yet we're just socializing realizing that change is coming and listening to our stakeholders and understanding their fears and their pain points so that we can then come back to them and say all right we heard you here's the adjustments we made to address your needs and I'm asking you to try this and we will iterate from there because some of these changes are pretty substantial for our businesses in the way they do their work today and uh change is hard for people

we need to learn to speak the language of the business security is incredibly bad at this security teams like to stand on a high and mighty pillar of we're not sales we don't we don't sell anything we're we're morally neutral uh we're we're the you know Paladin of uh safety and uh and so we roll our eyes at Revenue conversations at GTM uh at uh thinking about what what features move the needle in which segments and for a lot of security Engineers uh just like they don't like compliance they don't like industry standards I I've never met a security engineer that was like yeah we should do a nist CSF assessment and see where we

score in these different categories like that's always like the siso and the board that wants that but if if as a security engineer you don't start learning the language of the business and learning what motivates the board and the SE Suite to unlock resources for you you can't compel them and convince them and make the great argument of hey I know we were really concerned about our uh respond rating in that last assessment and uh I've been thinking about it and I think if we do this you know basically leverage your inner clickbait if we do this one crazy thing we can shift our score probably by four or five points you know tents of a point there and and

that would you know get us closer to Industry benchmarks or um you know thinking about yes I know this expanding this tool will cost us a million dollars a year to that tool provider and here's all the cost savings to the business in terms of engineering hours um you know risk reduction we're going to have better visibility to protect our customer data and so if you start learning how to communicate with these senior leaders you will uh get more of the resources you need to do the work you need to do to protect customers fundamentally our job is to enable customer trust yes we secure an application yes we secure an Enterprise but why do we do that we do that so that

customers can trust our company with their data and if they can't trust your company with their data they're not going to give you their money and then you don't have a job so your job isn't just secure the application securing the application is what you do on the journey to customer trust and so thinking about it from a perspective of is this building platform integrity and this comes back to why I'm saying get to know your customers understand their perspective advocate for their for them and think about what are the things we need to do to create that trustworthy platform um that will unlock Revenue that will pass vendor Security reviews when a customer is considering using our

solution um or or doing business with us they'll they'll have questions let's make it easy for them to get those answers and feel good about it so uh let's go make things better is my last slide and I did definitely miss a slide and this is incredibly awkward for for a keynote but I want to go back to it because it's important so something I meant to mention here um when I said that security Engineers aren't good at everything and I ranted about how like why are we having them right documentation I keep seeing organizations not hire security program managers or not hire technical writers or um not hire Community managers to to manage their bug Bounty and their

internal uh Champions program and uh The Fellowship of the Rings wasn't nine elves it wasn't nine Hobbits it wasn't nine humans and it wasn't nine dwarves you need diversity of thought diversity of experience and diversity of skill on your teams and you need to actually build teams not I have this technical engineering rock star we give them a problem they go solve it what happens when that technical engineering Rockstar gets sick or leaves the company and you haven't built a healthy strong team of people who help each other cover for each other address each other's gaps and strengths and weaknesses and so as an individual engineer or security program manager or security Community manager you don't

have a lot of control over this but if you are responsible for managing a team for influencing a business's hiring decisions think about diversifying your team portfolio I generally uh would suggest one program manager for each eight Engineers because that can in really really help in driving Business Solutions forward and driving the projects they're they're doing forward what typically happens is the engineering manager ends up working as a program manager [Music] aligning all the stakeholders coordinating all their Engineers but uh I would encourage you to diversify your team a bit and not just hire multiple cookie cutters of the same thing and then be surprised when uh your your documentation is not well done is late

is out of date you want that person whose passion is making sure the wiki is up to date and all the developers across the company know where to get training and and they're doing the assessment each year of this training has gotten out of date and we need to update it hey I was looking at the bug data that team keeps having a root cause of buffer overflow and here's the training I think we need to invite them to um and so there's a lot of things that Engineers aren't good at that we need to diversify and do more on okay clearly I had more to rant on that okay so now let's go make things better

let's go diversify our teams let's change the way we think about security let's start actually uh focusing on quality and customer trust versus just this cool new bug and think about how do we help our engineers make the right decision about links to click uh how do we help our customers make the right decision about uh data that they should be sending to our platform and and thinking about it from from the user perspective um sdl programs secure development life cycle programs Dev SEC Ops whatever you want to call them they've historically all been about tool and process here's the tool and follow this process and there's so little focus on the user experience in 2022 there's a

conference in Hawaii called Loom mosc and it's all focused on appac and specifically defense and Dino doovi uh gave one of the Keynotes there and he said sdl is dead Long Live secure developer experience and the secure developer experience is thinking about how do we make the ux better and how do we create these paved paths for our engineering partners for our uh employees for our customers where doing the secure thing is easy it's not hard it doesn't create friction and if your Dev team really really really wants to roll their own crypto like okay sure the security team can come in and help them understand why that's a terrible decision but you know they can they can come in and help but

why would they do that when they've got a really easy uh solution right there in front of them you're creating this paved path why would you go off-roading I mean okay we can help you go off-roading if it's really important to you but um it's going to be faster to just stay on the highway of secure development and so again everything that I'm talking about here is about how do we make that easier for the dev team and stop being friction and thorns on their side and and creating problems that they don't want to work with us every one of us has probably walked into a meeting with an engineering team and you can see the

defenses go up they expect you to be a problem they expect you to be argumentative and if you come from a culture where you're all about collaboration you're like hey I want to know about what you're working on and how can I help and all right have we thought about this or I can't tell you the number of times where I've had a team who's like you know we're not going to make SLA and I'm going to need an extension I'm like okay like let's talk about what's your remediation plan like this is this let's have the conversation I'm not going to like flip out about it let's talk about tradeoffs when do you think you could have it fixed is there a

mitigation that would be easy to implement in the short term and it will take a while for them to realize oh you're not one of those security people who's always coming in and making unrealistic demands and you know you're here to partner with me you want to actually consider how this affects my workflow and I'm not going to lie to you it's not all sunshine and rainbows there will still be those Dev teams that try to take advantage of you and be like yeah we'll fix this in 9 months and it's like m no that that one you can't fix in 9 months um because we've had an engineer who wasn't busy bug nannying like how many security Engineers spend

time chasing down product teams to be like no you have to fix this no you really have to fix this I swear you have to fix this here let me show you the repo um that's not a good use of their time and so if you can build a healthier mechanism for cleaning up that security debt by reframing the culture around quality now your security Engineers can come through and be like all right hey I really appreciate that you cleared up most of your security debt you're down to the last three bugs that we filed and you're asking for a business risk exception we took an in-depth look at them we had our vulnerability researcher

look at them and two of them actually sure no problem we'll sign off on the business risk they're really unlikely to get exploited the third one it's actually pretty trivial to exploit but we discovered you don't have to update the entire library and change your architecture this code snipp it right here this like one line of code if you go in and change that you've addressed the vulnerability and I can show you how and that changes your entire relationship with the dev team where you're enabling them and you're not just saying you have a bug go fix it so I know we can do this better uh I know we can reframe how security engages with

development um it'll take time and uh but we've got to do it or else we're going to keep getting the same results we've been getting and it's not going to change so or we could just go raise baby goats I mean that's cool too [Applause] [Music] thanks