AppSec as Glue: Building Partnerships to Scale Security

Uh so let's start off. Uh we've have uh Sarah Lou as your moderator and host. Like you'd give her a warm welcome, please. Thank you. Thank you. And across the uh the list here we have uh on the your left folks is Ariel Shin and we have uh Jacob Salassie. Thumbs down. Uh Makun Sarma. And last folks, Tad Whitaker. And let's give them our attention. Let's give them our presence. And uh take it away, Sarah. Great. Thank you so much for the introduction, Dana. So, welcome everyone. Thanks for being here for our panel to discuss AppSc as Glue, building partnerships to scale security. I'm really excited to be up here with this panel. They have some really great

success stories and insights on what has worked and what hasn't worked in their experience. So hopefully you all will have some actionable takeaways to bring back to your organizations and your application security teams. So uh Dana asked some questions to the audience already and I'm going to tack on some more. I'm interested to know who's here. So can you raise your hand if you work for what you would consider a startup or preipo? Wow. Okay. So that's probably like a third to a half. Pretty good amount. Um, raise your hand if you work for a larger company like post startup or you know 500,000 plus. Great. Um, and then two more questions. Who here is an

appseac or in something similar? Awesome. Love to see all the apps folks here. And then who here is not an appseac but you're interested in building out partnerships with ABSSEAC? Love to see it. Okay, we got a good ratio here. All right. Um so a brief over overview of our panel agenda. We are going to start with our classic introductions and an icebreaker and then transition into some uh questions we have prepared. And Dana, if you could help let us know if there's any new questions coming in from the audience. We would love to bring those in as well. Um but yeah, so I'll quickly knock out my intro. Nice to meet you all. I am

Sarah Louu. I am a builder at heart uh moderating this panel. I come from a software engineering background but transitioned to into apps like when I first had a runin with security at some point in my career. So I've worked in several industries finance, entertainment, telecom and I currently run the pentesting and bug bug bounty programs at Twilio. So turning it over to the panel if we want to start I guess closest to me. Ariel, can you please give yourself a short introduction and name a person that you admire most in security and why? Oops. Hello. Maybe now. Is it on? Hello. Okay, it's working. Yeah. Hi everyone. My name is Ariel and I currently lead an appse team

focused on threat modeling, developer security training, secure defaults, and risk scoring. Prior to this role, I led the appsac team at Twilio. Before that, I started off as a pentester, made my way into appsac since I found that my vulnerabilities would get reported but never fixed. So, I've been enjoying the apps space ever since. My focus has been scaling apps and reducing engineering toil. Hi, I'm Jacob Salassie. Uh, I've been on IRC since age 10. Uh, I didn't go to college. I was a network engineer. I was a software engineer. I got into security. I led product security at Snowflake for five years and then I left to do startup type things, just mess with stuff with friends. So that's kind

of what I've been doing lately. Yeah. Hey everyone. All right. Hey everyone. Uh Mun Sharma. Uh I lead product security at uh Chime. Pretty much do anything security really. Not really specialize into one domain or the other. um pretty much open to doing anything. Um joined came into security as a software developer, moved into traditional apps, been doing that for about 10 years now. Um I love it. Uh hey Tad Whitaker. Um I'm the staff product security engineer at engine.com. Uh before that I was the first security person at CircleCI and I led that for about five and a half years. went and did something similar at Apollo GraphQL and then outside my day job stuff. Um I

used to run the Bay Area OAS meetups for about five and a half years and I'm a co-founder of the day of security that helps women get into security careers. Um there's a booth downstairs you should definitely go visit it. And Tad, starting from your side, can you share who you admire most in security and why you admire them? The person that comes to mind initially is Jamie Finnegan at Hashi Corp. Um I went to engine to be a product security engineer kind of because I hadn't like done that and it was interesting but he's the product security lead and has been at Hashi Corp. And so I've hassled him for about 50 me Q&A meetings over the past 18

months. He's just really knows what he's doing and super open to helping me. A kind uh this is a tough one. I have too many of them. Um but I think the one that the one that made the most impact I would say is um Jeff Bridto who's actually also my boss but I've been working with them for like the last seven eight years now. And I think a lot of security is actually how much charisma also you bring to the table. It's not just purely your technical jobs and skills that's you can get those but to actually make an impact in your organization or to the community. I think it's also a lot about

how much can you influence um people and I think I've learned a lot from him on that. Uh so I don't really have heroes so I'll go ahead and say me first but besides me uh I'll just mention Ryan McGeehan who I don't know at all Magoo but wrote a lot about quantitative risk. And if I thought about the person who precipitated like the longest amount of thinking on a topic I've ever done, it's quantitative risk and it started with stuff I read from Magoo. So there's that. Uh person I look up to, I can actually see her right in the eyes right now is Colleen Kulage who was the CISO at Twilio segment and has really

inspired me to be an empathetic leader and to think about all different types of viewpoints and really puts her people first more than anything else. Awesome. Thank you. And I'm gonna just call out our missing panelist here, Jieven. He is on the signs that you might see. Um, he unfortunately wasn't able to make it to SF today, but we have Tad. Shout out to Tad who was so willing to sub in very last minute. We are very excited to have him on his panel with his experience and expertise in this subject. So, um, we're here to talk about building partnerships across the organization. And I want to call back to our title apps as glue. So

I want to hear just to set the stage from everyone on this panel. What does it mean to be glue and can you give an example and Ariel if we could start on your side? Yes. Uh so I interpreted glue glue as acting as organizational glue with engineering and your peer security teams. uh security teams can be siloed from one another and we can also be siloed from engineering especially if we don't sit under the CTO and apps teams are one of those teams that can act as that glue and can become the face of security for many folks who don't often interact with security. So the way I thought of it is that we have a lot of

soft power and political capital that helps us get security work done without using policies or brute force. At times we use our pre-existing relationships and a common example is through security partners where we partner with an engineering team to get security work done by giving them a lot more focus so that we're focusing on high-risk issues within a certain focus area. Yeah, I would say something similar to that and I think maybe you made the statement which is like security is accountable but not responsible. Uh but I think you can maybe just take like not a bigger view but I think about every company has a dysfunction like every organization especially everyone who works at a startup. I'm sure you're all

living some version of organizational dysfunction and so maybe there's two things I'll say about that. One is like security is often in a position to witness organizational dysfunction and be inhibited by it which then leads to two like you're in a position to leverage I'll just say remediating organizational dysfunction as a way to gain political capital not in a perverse way like your help in bringing the business together is ultimately key to your success and something you will be uniquely in a position to do. Okay. Um, I noticed a bunch of people raised their hands, probably like half that you're preipo or startups. And that's definitely my area of expertise. I've never actually even worked for a public

company. And so, um, really speaking to that audience, um, when I think about what glue is, it's, uh, you know, as it relates to ABSSE, I'm always having to start it. And so to a huge degree, what glue means is just going in and listening to what's already there. Um we have Artifactory, but who's using it? Like should we cancel it or is that actually something that we can, you know, that a leadership person really wants to focus on or some form of that? But it really is to me going and listening and probably wind up doing a lot of a lot of program management and vendors talking to vendors and things like that. But it's really starts with

me being a listener and trying to insert myself into wants and needs that exist and trying to uh create some cohesion there. Yeah, I think it's a combination of almost everything what everyone said. The thing I'll add is um it's a two-way uh relationship. The way I look at it, I think even software engineers or your other counterparts of the company can also act as glue to your security program. So I think it's to me glue is whatever it takes to make that um particular risk uh go away or reduce the impact of that ever being um action on. So um clarifying the risks to people um who are at a decision-m power uh in a

very pragmatic way being able to tell them what are the immediate self things that you could do to bring this down. Um all that stuff is usually not really really considered as appsac function in many traditional companies. It's considered as maybe a grc function or um something else. But to me the appsec glue function over here is the person who takes it about the just finding that vulnerability like what does it mean to go fix it? Can we build a quick library? Can we introduce something that can make it easier for people to onboard to that that additional piece is what I would call as glue. So Tad, you mentioned that you mostly have experience with startups and McCun,

I know you've mentioned in a previous conversation that you've been the first security hire at other companies or at least the first apps hire. So I'm wondering at least from you two, but if other people have thoughts to chime in as well, what is the first partnership you prioritized at building at your organization and why? I can go. Um I think it really depends on what the business is I would say. Um if the business is building a product or an application and then that's the biggest asset or the risk for that company then doing that stuff is the most important thing for that um for that role. So uh for the companies I've been I've always been in the product

side or building um both fintech companies. So um building up relationships with our uh DevOps or um building relationships with the team that's maintaining uh frameworks or anything that way that that way you can amplify your um impact not just for one team but to all the teams that are using the same set of um underlying infrastructure or um frameworks. That's the kind of way I've approached it. Mine would be um inevitably when when I when I get hired, I wind up interviewing with a CTO or somebody like that. And so if I've gotten hired, I've already got alignment with that person. Like, you know, they wanted me to lay out a program, an idea

about how to build this thing out. And I either did or didn't pass that. And so I would say after day one, it's really trying to find like a chief architect like at engine.com who wasn't on my interview panel, but nothing happens without that guy's um sign off. And so really uh that's the number one for me because all software gets shipped through him and what he prioritizes. And so, um, that's that getting alignment with him on whether or not we do a bug bounty program or if we don't do it, what are we going to focus on with the money and the time and just making sure that my idea aligns with his needs to

ship product features. And um, I'm going to call you Ariel since I we've worked together for a little bit. So I know maybe you have an answer for this, but what team ended up being a surprisingly strong security partner and how did that partnership develop? Um, so I worked on a project called democratized vulnerability management where we worked on centralizing our vulnerability management process. And so this affected all of engineering. Um, and one of the things we had sought were for some strong cheerleaders that we felt would really help propel this program forward and also identify opportunities for us. Um, and we had one really great partner that I called in my talk, Bob. Uh, who

really just went above and beyond in championing our process. He really liked what we were doing with this program and partnered with us to create dashboards, not only for his team to ingest these vulnerabilities and prioritize them and talk during monthly business reviews, but also for all of engineering. He had catered it for engineering and became our engineering voice. And we didn't even have to ask him. This was something where he came to a meeting and said, "Look what I've made. I'm really really excited about this program and so I really wanted to contribute. And for the rest of the panel, have you had similar experiences like this or any other surprising relationships have built over time? Uh maybe I'll mention

one. I I guess it's it's not unusual to me, but uh internal audit. I don't know if how many of you are at a company that has an internal audit function, but you probably live in a world where security is already something people don't want to do. If they really don't want to do something, it's also internal audit and face that team. Um, but you also know that internal audit usually reports to CFO or those types of organizations and usually wields a lot of power. And so something uh and again I said I don't have any heroes but Mario Dwarte is a guy I worked with at Snowflake uh my boss. He was the CISO there. And

something you always told us was you know whoever you perceive as your enemy that's how you that's who you have to bring in. And we brought IIA in and then we put IIA to work on stuff that we didn't like and it was like running around with a flaming sword. a really great relationship to have us and IA and us feeding IIA things rather than them coming and saying look what we found that you're doing wrong or whatever. We worked together, not in a weird way, but it just it worked out. We started out kind of mentally as adversaries and I think when we lowered our egos and came together, uh it became like super

powerful when you really wanted to get something done at at a large company. I like how you mentioned flaming sword because I feel like it it fits in with the theme. So great visualization. Um so while you were speaking Jacob I'm going to pick on you like you've been at or you were at Snowflake and you saw the engineering or grow from 100 engineers to a thousand engineers. So that's 10x growth like crazy. So wondering what your thoughts on and and how you've seen the evolution of security partnership look over time and over that scale. Yeah. So when I think about probably security partnership, you know, there's a couple of ways to think about it, but when I

think about like at the ground level, one thing we learned was like, you know, initially it's good to be really hands-on. It's later it's good to get every developer doing something, but later than that as you get beyond 500, uh, you know, I really kind of learned, especially toward the end of my time at Snowflake, that it's actually not a great idea to ask developers to know more and more security. And there's like this arc. It goes up, it peaks, and then it needs to drop precipitously down as you get more engineers. And you have to like stop teaching them security and start delivering security in a This sounds like pretty cliche, but like you

have to bake it into the platform. And if you're not able to do that, you're going to struggle uh to scale teaching a thousand people, right? There's no there's no like homogeneous way that a thousand people behave. like they all behave differently and some of them may appear actively malicious to your program but like you just have to deal with that and it becomes something you have to ask them to think less about. So that's maybe the dimension that stands out to me most from from my experience was how my view on what you expect developers to do changed with scale. At what number would you say that that arc peaked? F in hindsight I can say it was

probably at 500 like three to 500 people we should have been already thinking about what the next generation was going to look like and instead like we figured it out but we figured it out after we had already gotten there. Yeah. So you mentioned platform and and kind of throwing a buzzword in here is secure defaults. So I feel like that is really maybe trendy is not the right word for it but at some point you want to be working with platform teams and building insecure security defaults with them. Um but I know we've talked about like some challenges in the past between a group of us. So I wanted I want to hear more about like how that process

went building these security defaults at your organizations and how do you get it funded? Yeah. Um maybe I'll I'll continue. Sorry. So I'll just say something I'd love for other folks to say something. So security defaults uh especially at scale my experience is organizations are not going to accept platforms or software from security by default like they expect those things from platform teams and they expect those things from infrastructure teams. And so there's kind of like two roads. There's probably more than two. It's never just two, but there's two I can think of which is like you either figure that game out and you're influencing those teams road maps, which is one game you can play.

Another way is to become those teams and hire software engineers and hire infrastructure engineers who can ship production code. But I think the mistake a lot of teams are in like an anti-goldilock zone where they're not really software engineers, but they're smart and they can hack stuff together. And I think like actually it's not a good idea to ship that. It's not a good idea to predicate secure defaults on that. So you either need to become a credible software and infrastructure engineering team or you better really focus on those relationships and not just hey we're friends but like hey we have a joint road map and I see my stuff in your quarterly plans. I will say when

you do hire out that security platform engineering team or whatever that name is, you still have your classic appsc team that still needs to then act as the glue between those teams and identifying those priorities, working with those developers and sharing feedback on those platform features or what they like to see, what they don't like to see. Um, as I'm currently rolling out a secure default program, it's been we've been heavily rellyant on the platform teams and working with them to get feedback on how do we get adoption to 100%. Um, and is that even our goal or should we be looking at other opportunities so that we actually do drive down risk in sustainable ways?

Yeah, I think it's an extremely hard problem that people don't talk about more. Um I think it's kind of thrown around that security falls and um shift left and all of that is like things that people should do and go look at but some of these projects take multiple years at minimum like I I was like you think of things like um we spoke about this last year which is like services service authentication and authorization it's a simple problem we have hundreds of services they need to authenticate and authorize each other that platform took us at least like three three and a half years to build and it out and um we had the buy in from engineering, we had the

buyin from platform but to get it out so that it's secure default it's a hard problem it there are several languages there are different text stacks there are things that you need to like handle and now you are at a point where you need to like manage all those use cases so back to what Jacobs was saying you like we became that second use case which is like we hired software engineers that built software components and deployed real production stuff but it came with a cost of us having to maintain and manage that and also back to what Ariel was saying is how do we know when something when all our assumptions are failing right we also

have to like continuously test our own platform and our own solution and also look for um issues on how some of that can go wrong so I think security falls should not be looked at complete binary like it's there or not there I think it's a whole journey and um you should definitely strive to getting there but um there are a lot of smaller things that you can start doing that will have a lot more impact while you work on those. That's kind of how I look at it. Mine would be uh I tend to focus on two things right when I get there. One is just uh hardening GitHub access branch protection um just really cleaning up

GitHub and how that works for engineers. And then the second one is whatever cloud provider. So it could be AWS or GCP or whatever. But focusing on those two, everybody has some want or wish or need on those two. Um, you know, and then I also tend to go in and turn off a lot of stuff and just quit paying those bills. Like I don't know, or somebody previously said, "Oh, wouldn't it be great if we had this and they got it approved because there wasn't a rigorous budget program or something." And so going in and just turning that off and that off and that off, we'll get to it later. let it hit the floor. Let's focus

on GitHub and actually where engineers spend the most of their time, which is, you know, GitHub and the in the cloud. Yeah, definitely agree. Meet where the engineers are and keeping keeping it simple, I guess. Also, I love turning things off as some people in my company may know. Um, so we've talked a lot about partnering with platform teams, with engineers. Um, Jacob's mentioned the audit team as well. I was wondering if uh we could speak a little bit more towards partnering with teams within security because appsite is just one group within security. I think there are many silos that live within security as well. So if we could hear from the panel if you have any stories of partnering

with another team within security. We would love to hear your experience. Right. I do. Uh and well, he's sitting right there. Uh so something something we thought a lot about and I think I don't know. I think this is true. I think uh I was having this conversation with Michaela. He's right there. What I said to him was I think TDIR threat detection, incident response, red team like they all sit over here and product sits over here and like it's like the allegory of the cave. like the people in TD and I are like have a some idea shadows on on the wall about what happens in product but they don't have any direct knowledge of it

and so I think that heavily impacts like the quality of detections that get written and you you tend toward like this least common denominator of what incident response and detection can be. That's not going to be true for everyone. I'm sure some of you are living in some utopia. But what we realized cuz we were really close and had worked together before was like well this is stupid. Wouldn't it be better if like there was a full a full loop and like every time we did a threat model or every time we did a risk assessment that like those were going to turn into detections or rather how could you answer the question of you know what is

the detection that needs to be written for that uh for that particular feature which I I don't feel like a lot of teams even ask that I think like features happen and detections and other things happen. So I think something that was really important for us was not just us being friends but like having a structured system like a process that pulled all these stakeholders in and had them more or less involved because I don't think you do threat detections for every single feature but you definitely do them for strategic investments and things like that. So conceiving of that and formalizing a program around how am I going to get these people in, how are

we going to get the detections written, how are we going to get the response plans written, and what is the red team going to do uh on purpose was was something that was uh highly valuable. Something I've strived to do is with threat modeling connect with the pen testing team, connect with our threat detection team, connect with our compliance team when needed. Uh, but it's something we haven't perfectly figured out yet because even a threat model alone can take up to a month to find the right resourcing, timing, and to get it done. Um, and someone had told me something that I thought was really great in that uh she currently manages uh two different security teams, an

AppSsec team and a threat detection team. She said she feels like she's code switching. If you talk to an appsec person, they want to hop on a call. They're super friendly. They want to chat with you. They want to get to know you. You talk to a threat detection team and they're like, "No, no, back room only. We want to have our secret jokes. we do not want to be on those calls with you. And I found that really relevant because uh when we try to connect all these different partners with our threat modeling process, we don't all have the same way of trying to communicate with one another. At the end of the day, we

need to identify risks and those mitigations or detections that we're going to build, but the way we go about doing it can differ on that team and I still yet to find what type of processor program would allow us to merge all of those things. But generally we've started with you start with threat modeling and then we'll kick off to our partners. Whether that's a Slack message request for our cloud security team to look at our Terraform rules or with our threat detection team to look at logging and monitoring some additional detections. We're still trying to find that balance but we're at least having the conversation together which is more of an appseack thing than I would say

any other type of team. Um mine would be threat detection for sure. Um, again, kind of being small. Um, threat detection is the one that's kind of going out and and investigating the the known gremlins that everybody's like, ugh, you don't want to know about that thing over there. Um, but so partnering with them to really uh prioritize what appsec work we wind up um we wind up tackling first um and being okay letting other things come later. Yeah, I think I've had quite a bit of different kind of stories over here than some of the what we just heard over here. Um, we've actually had quite a bit of luck with internal teams talking to

each other, working on different projects um together and things in that sense. So um couple of examples I think is we've um opened up the um writing going and writing our own alerts and prediction to anyone not just security team including the um engineering teams if they need to and if they want to uh there times and they have come back to us and said hey we want to know when someone is executing into our container or our pod and we need to get detection and alerts and that and we're like okay here's a runbooking here go write it up. Um but I think the one thing which we've uh kind of have always kind of struggled

is how do we we try to like be very pragmatic and to the point to the extent where if sometimes we don't know how to solve for it we don't even bring it up to the engineering team or the different domain owner I'm like no actually that's where we kind of draw the line because sometimes we may not be not know everything but at the same time it's our job to like tell them what the risk is and they'll either like accept it or work with us or something but I think sometimes it's been a very emotional conversation or um very angry conversation is the way I would put it uh is that people are like why do we

bring this to the engineering team if we don't know how to do it ourselves and I'm like it doesn't matter we got you got to present the risk to them so that's some of the challenges that we still kind of need to face in general but yeah in general if you open up the platform for people to um collaborate um I think it works out well for us so I want to bring up one more team before we kind of pivot into like more leadership influence questions. Uh we talked a lot about security teams and non-security teams. I'm wondering if uh any of you all have experience with working on the business side. So like people who sit on

the edge who talk to customers directly at my organization there's a team specifically under security who works with customers. So um very interested in hearing how working with these teams might have helped influence how product ship like have you talked directly to customers before? Yes. So I I think uh we we have a security engineering team that we call uh that also builds stuff for our product. Um one of the few things that we build a little security center um ability to like do pass keys and all that kind of stuff. So I think this is really good because it gives uh the team a good visibility into what the product development life cycle is and

also it builds credibility for um the other teams in the product and engineering teams that the security team knows what they're doing. They're able to ship stuff that is actually relevant. Um a lot of that is actually uh we use a lot of NPS data. We are a B2C company. So we kind of look at what what's the biggest reasons why people don't like our product and why are they leaving and if there are any security related stuff there we kind of prioritize those from our angle working with product. So that's been pretty helpful for us. How about from the other people on like B to B2B companies? Um when I was so I got my first security

job at CircleCI just because I was good talking to customers. Um so like I was a boot camp grad. this is a long time ago. Got the entrylevel support job and I just answered all the security questionnaires cuz I was curious what scared people and at a certain point I knew more than the CTO and the head of S about all that stuff. And so there were just a couple of last minute oh my god we've got this huge customer on at 7 a.m. tomorrow morning. We need you to get on there and reassure them what we're doing. And so I just became that person and then at a certain point they just had to give me

the title security engineer. Um but so a lot of what I wound up doing was customerf facing initially but then listening to what those customers needed to sign the six figure or seven figure deal. Um I was the person that went back to the CTO or the head of product and gave him the hour and a half long um demo of it or I guess you know just like outlaid everything and then I was the one that wound up kind of managing that customer relationship you know for a while. I'm curious yours from Snowflake perspective or um Julia or any of your Yeah. Uh when I was at one medical um I worked really closely with the mobile

teams and we had a customer audit our mobile apps. Um and those were a lot of early morning calls which we weren't used to before. It was like no meetings before 10:00 a.m. but for a customer you'll wake up at any hour. Um, and it was a lot of back and forth and it was really nice that I had uh worked very closely with the mobile team because it meant only I had to wake up early in the morning, not the mobile team and I could be that interface between the teams acting as that glue representing the mobile team, especially when there were some like requests like disable copy and paste. And it's like, well, how will

folks use password managers if we disable copy paste? um and kind of working with these like medium-term solutions uh with customers that oftentimes have like very like strict checklists or requirements that they want. Um but understanding that they're not our only customer. So balancing wanting to enable the business but also like security requirements that make sense for our apps. Uh I'll quickly mention three three ways. So the first way was security on Snowflake. So some of us Snowflake had a product called Snow Alert which was like a very early proto sim. There's much more stuff going on there now, but in the very early days like some folks would be on the phone with customers like more or less helping

sell that into accounts and then get getting them stood up. Uh the second ways the second and third ways are a little bit more aligned with what you've heard here. There's of course like we had a GRC team who frontended customer conversations about like hey how secure is the platform? So the relationship with them was mostly about like hey what are the findings you're going to send me that you know we led product security so most of that came to us because we're selling a product. So, it's almost like a pentest type relationship where it's like, "Hey, we need to find dupes. We've already answered this. How do we efficiently make sure that the questions

get answered and the unanswered questions make it to us?" Um, and then the third way of course is like talking customers off a ledge or whatever, right? We also had to just meet especially in the leadership roles like any big deal that had a question about the product like they want to meet the CISO. Mario being a good guy is like, "Let me bring the people who actually know how this works to the call." Uh so we would often be on those calls. I mean for my view and what I write on my resume is we help close those deals because we did. Great. So question from the audience like many of you have leadership

experience whether managers, directors or in a in a past role something like that. So you have your own circle of influence and then like influence that you need to go up or to the side. Um I think like maybe the T is what I've heard it being called. So, what metrics have you found most most effective for demonstrating the business value of ABSSEAC partnerships to leadership? And this is from JJ. This is my favorite question because it's really really hard especially when you want to represent your partnership work. Um, it's really hard. I think there's like an idiom in a Netflix scaling apps blog that said wins are far and few in between. So it becomes really

difficult to measure that partnerships are successful in ways that enable the business to say let's keep funding this. Oftent times rely on qualitative data where a team says I feel really good that this partnership is working and it's leading to a lot of great results but that's just a qualitative statement that doesn't really say if risk has been reduced. Um you can start to count the number of threat models done but that can really be difficult. So I do feel that every organization has a balance where they look to see kind of what were the projects done and what are the outputs of that and it's not necessarily a total number of what's the percentage

increase of uh risks reduced from the time you embedded with that team versus you didn't. They'll tend to look at the types of projects you worked chat with the engineers to kind of see that appeal. Uh but it's no way perfect. So if folks kind of have a better idea, I'd be interested to hear because I've always really struggled in balancing the qualitative and the quantitative to really convince folks that this is something that they should continue to invest in. Uh so from our engineering partners, something that somewhat surprised me, they were often interested in like can we show the MTR of bug went down like MTR of vulnerability? They cared about that. They cared about like the number

of findings per capita. So per capita I think is something that's interesting because it deals with this problem of I'm always hiring more people. So I guess the TLDDR from their from their perspective is they were interested in how it made their workflow go faster and they didn't really care if we told them like how many vans were in SLA for example which is like the classic uh uh metric. They didn't care about that. Um but the second thing we did and the most powerful thing we did and it just was a function of where I was in the organization and who we dealt with was like well you know we were delivering the set of risks to the board and the

ELT and that is how we communicated like what the success was and it was you know the percent decrease in this particular scenario's likelihood over time. Uh and I don't know I'll just say it like if you're influencing at that level great like do a good job of it because everything is much easier when the entire seuite agrees with you. Yeah, I think uh for all the operational security stuff, we've kind of gified that. We've built this platform called Monle, which we've kind of spoken quite a bit and written about, which basically gives every security team and a security every service that's being deployed in our ecosystem a security grade. And the requirement is you got to be have a B or

above. If you're doing that, you're doing fine from a security perspective. And that's one metric that every time when we go to engineering metrics or things in that sense that we present to say how many teams are doing well and how many aren't and we usually get good response for the teams that are not doing well to like go back and do it well. uh more at the leadership level I think just the security uh residual risk or the risk register like how what what's are the big blocks that we actually made uh we moved quarter to quarter and what does it mean um a lot of times people also want to know how much of their time is their team's time

is spent doing security work so we have kind of even measured that that's been helpful uh to say the security KTLO work um that's been if you if you present that and tell them that this is this has gone up or this has gone down for these reasons that's a pretty good metric for them also. So those have been helpful for us. Quick question for you, Mcken. If you were to rebuild another security team, would you ask your team to start building a platform to capture metrics on um how effective your partnership work is or would you start diving into a partnership immediately? Both. Okay. Yeah, that's kind of how we did it at Chime here also. It's like we

the day the first day when we started we started building that platform along with doing the partnerships. Nice. Uh mine would be um beyond looking at CVES and vulnerabilities and dashboards. Um I'd always go back and look at all the incident postmortems and look at the root appsec causes and try and find themes. Um seven out of the last 10 were uh a result of like a leaked secret. So prioritizing that and then actually being able to show that we're burning that risk down to stop a whole class of not only just like external vulnerabilities but all like the top 10 engineers at the company have to drop everything for three days and rotate this or whatever. Um every leader likes

to see that that stuff is being taken care of. So we have just a couple minutes left. I'm going to ask one more question to sort of wrap this up, but um I feel like we've heard a lot about success stories and building these relationships, but as Clint has shared in the keynote, we also want to talk about failures. So, I want to hear from this panel, what is the hardest lesson you've learned about building security partnerships? Okay, there's not one way to do it. The thing that I messed up and we messed up so many times was either we went to the top, in other words, their managers or their never go to the middle manager.

Pro tip, don't just go to the VP or whoever or go to the engineers. Forget the rest. But it's either doing one or the other and not both and then not overcommunicating. And so you end up creating these situations like we partnered with a DevX team about program analysis. We were all hyped up, a working group, a bunch of engineers. Their leadership was not into it at all. Which means that as soon as priority shifted, like duh, no one was interested in this anymore because we didn't do enough work with the leadership saying like, "Hey, this is what we're doing with your engineers and this is why we think it's important and why they think

it's important." One quick thing for me is um I kept going to these meetings with VPs, engineers, directors saying that, "Hey, this is the stuff that you need to care about." But I realized that didn't work. So I changed the question to what do you worry about from a security perspective? And that changed the whole narrative. So that worked well for me. My big one is around security education, assuming that all software engineers want to know about security um and are interested in it. Um I think it's, you know, pretty similar to kind of what he was saying. Um I just don't assume that anymore and I just go forward with my job as is.

Uh mine is establish early relationships. You never know when you're going to rely on them. And start making your successes visible and loud. You want to continue to fund that partnership program. So any way that you figure out to measure your success, show it outwardly. I'm just going to reiterate that last one. Something we were really terrible at was celebrating what we did or just talking about it. We would often see other people talking and have a cynical view of that, but like in hindsight, I'm like, "You're so stupid. Just talk about what you did un unashamedly." Yeah. The Slack channel posts or my company calls them hoots, right? We got to celebrate ourselves if no one else

going to do that for us. That's right. Especially security teams. Okay. Any closing thoughts from the panel here? Great. Well, thank you panelists for your time and thank you everyone who joined us today. Really appreciated you sticking around listening to us talk and yeah, round of applause to our panelists and thank you Dana for volunteering. Yeah.