BSidesSF 2026 - Running an efficient bug bounty program and PSIRT... (Garrett McNamara, Jeff Guerra)

We have a uh two fabulous presenters here for you. We have Garrett Mcnamara and Jeff Guerrera running on running an effective bug bounty program and psert function. Take it away folks. >> Thank you. Thank you. Thank you very much. Thanks for being here. Um just to get started, you know, keep your hands up. I'm going ask some questions. It'll be short. Um, who here are, you know, is like a bounty manager at your organization or if you support a bounty program. Awesome. Keep them up. Keep them up. How many are bug hunters? Trying to get an idea of the other side. Nice. Nice. Um, and then who's here to sing uh, Since You've Been Gone by Klay

Kelly Clarkson for like 45 minutes. There you go. Yep. Yep. Yep. Thank you. Thank you. All right, let's get into it. Uh, who are we? Why are we here? Um, my name is Jeff. I'm a senior security engineer at One Password. Um I just wrapped up five years at GitHub supporting their bug bounty and vault management program. Um I've dabbled in appsac bounty programs and uh VM programs. I love chaos and to balance that uh because I work in security on the weekends. I'm a hardcore gardener. Um and I don't touch my computer until Monday. So um that's me. >> That's awesome. Thank you. I'm Garrett McNamera. I'm senior manager of product security adversary management at Service

Now. And that's a mouthful. So it covers a number of focus areas uh really roughly offensive security, defensive uh risk management, bug bounty, mergers and acquisitions and remediation. And if you send me a better photo, I will give you my drink ticket because that one's really old. All right, so we know fires are inevitable and really what we're trying to do here is try to help you make them more manageable, right? We know we're never going to get to zero fires. So today we're going to cover some things and give you a couple takeaway resources as well just you know to be helpful. And so first we're going to cover automations and incident response processes. Then we're going to get into

exciting metrics the bounty community team maturity uh program maturity and those takeaway resources. All right. So there's a lot of integrations you could do a lot of automation opportunities. Um, I like to focus on Slack integrations the most. Um, I really like to do Slack chat op automation. Um, thanks to my time at GitHub, um, I got to be exposed to a lot of like really cool Slat Slack chat op automation. Um, it could really help with streamlining your bug bounty v vulnerability tracking and it can give you like automated real-time notification alerts for just like things that you find interesting at your company, right? But like if you want to know when each report comes in, that's

an easy integration, right? Um when you want updates on, you know, reports that are already in your in your queue. Um another another checkbox, right? So um bots can really ex like assist in triaging out to resource owners auto assigning severity. Um not like by itself, right? like, hey, with a chat op, I can go ahead and kick off tracking a vulnerability with a specific severity um into whatever tracking system you use. Um, and you know, escalating issues with ease. Uh, you can do this with Slack web hooks by creating like a custom internal app. You could use Tines if you're a Times user. Um, I I love Tines and you a lot more, right? It's

also important to note that like Book Bounty platforms offer integrations with many tools and like mixing this with a custom app really leverages like one the platform's API and then it gives you like the best results in my personal opinion. Um, for more information of what this can look like from like beginning to end, like please check out a talk I did in uh Bside San Francisco uh 2024 2023 called Life of a Bug at GitHub um with a colleague and uh it shows you like the whole PERT and bounty program from like when you get a report to all the way when you like you know resolve it in your platform. Um so now

Garrett's going to talk about his portion. >> Cool. So service now is a platform and on top of that we built our own application because we didn't have one that already existed that really was suiting our needs. Uh what we by I should back up that in 2022 when I joined there wasn't a team um and there wasn't you know a very formal program. So we needed a few things. One we needed to guide people through incident response. A lot of our team had never been on a PERT. So that you know that was new and we needed also a place to save knowledge as far as like what has gone on throughout the course of an

incident so we can go back and and learn about it. The very first thing we did was we integrated with our bounty vendors API uh a two-way integration. So the existing one that was out there I think is just like a one-way and we're like hey we want to be able to push changes as well. So we have a two-way integration. Uh there's a number of internal tracking ticket tracking systems as well and so we integrated with those. Each of those has a different purpose. Um, but we needed a centralized place. And then we also actually integrated with a patch level monitoring system. And I need a better name for it, but long story short, for a

given issue, we're able to go and look across the customer base because we we host nearly all of our customers and say for a given issue, what's the patch adoption rate? You know, how how exposed how much residual exposure do we have for, you know, a problem that we know about? And then we also capture all that work that went into the response effort. Um that helps us get better. And again, we didn't have an app already that was meeting the needs to store like root cause analysis notes. Uh code samples, whether it's exploit or snippets of vulnerable code, uh draft disclosure communications if you do CVES and security advisories, and then even like upstream improvement opportunities to go

talk to other parts of the product security department to say, "Hey, here's some ideas for some improvements that, you know, we should put in place." Cool. All right, let's get into a three-part bit about uh actual response. So, when I mentioned in 2022, we didn't have a PERT, I got brought on to to start it. And uh the very first thing um I had already known about from a prior PERT experience was first.org has a services framework just for PERTS. They break down here's all the things that PERTs can do and they have different maturity levels. Like level one is hey, you're just getting started. Two and then three is very comprehensive. It's like, hey, here's a universe of things

your team can do. So, but the very first thing we did was we said, all right, we need some way to uh organize our incident response. And first.org, their their services framework will break it down into discovery, triage, remediation, and then disclosures and education. So, if you don't know what any of those are, discovery is confirming that the vulnerability is valid and it's reproducible. You know, it's not a false positive. Uh triage is trying to figure out where that vulnerability is in code. Like why did this happen? Is it more severe than we initially realized? Do we think this vulnerability is elsewhere as well and needs to be run down? And what's like the best way to go about remediating it

with the least amount of chance of like a bypass to a fix and then you know having to do a second fix. Uh the remediation stage is actually like all right, what is um what are the patches like a near-term uh a very quick what can we do about this right now? um does this need to be a multi-art fix once you get into disclosures depends on your organization if you're doing them but that would be your CVE and security advisories and then education is kind of that feedback loop back to your uh software secure software development life cycle. So one fun thing you might think about like hey if I were to ask you what's the status of an incident

you'd be like hey it's one of those things you just mentioned it sounds like a silly question we realized with complex incidents it it became a difficult problem where we're like all right how do we summarize where this incidents at because we know about the one reported issue uh the one that has public knowledge and then maybe there's some like derived concerns as well and so what we can do is we can say hey for that like acute issue that mainline line problem. Here's the phase, but there may be other work streams like other research going on or variants you want to run down. So, that was kind of funny where I was like, hey,

we had to go back to square one. Um, and then once you're in your triage stage, you may have some suspicions that the reported issue is replicated elsewhere due to code copying, some pattern, whatever it is. And that's where we kind of depending on Spidey senses um a lot of times say hey let's do some variant hunting or datadriven variant analysis on this. We think this exists elsewhere. We'll save everyone a ton of time if we just go ahead and fix these at the same time. So one thing you might be curious about I was curious about is whether the organization is um well whether the bounty program is helping to reduce like a stack of legacy

longexisting vulnerabilities um or is the bounty program serving to find vulnerabilities about as fast as they're getting introduced and is this just going to be like a continuous thing. Um so if you want to in kind of in a blameless approach uh look at commit history is is one indicator to say like all right some of these vulnerabilities they predate like serious investment into appsac uh activities so makes sense. Um, but that's that's one indicator where we go back and we say, you know, are the uh are these new issues or or old? And then another potential little fun exercise is if you keep your internal version of an OASP top 10 particular to your product,

is your bounty program burning down classes of vulnerabilities where, you know, you may say, "All right, let's let's try to tackle what's the most prevalent one over the years. Have you been able to um to burn that down and now you have a new new number one?" So, and after select incidents, uh, we take a moment to do two types of reflection for feedback opportunities. Some people may combine these into one, but first we've got a PI or post incident review. And that's that's all about the response mechanism. So, how well did the organization, including my team, work with others to quickly uh identify, confirm, and remediate an issue? And it's just all about the response

activities. The post-mortem though is what parts of a system broke to allow this vulnerability to happen in code. And so for those like especially large incidents that have to do with the product. Um we'll dive into that and people want questions like how did this happen? Are we in any better state today than we were yesterday? Like could this issue come back? Uh those kind of questions are you know we need answers like right away. So when you're doing these, as as fun as it is to try to like roast another team, um please take a note from Google's SR book on uh blameless approaches. So they they think that everyone has like a good intention

going into it. They've operated with the best information they had at the time and they made, you know, the best decision they could. But the blameless approach is all about uh you know, the person isn't necessarily the issue, but what part of a system broke down to to allow this to happen? All right. So, let's talk about part two focusing on the uh secure software development life cycle. Um, you know, let's talk a little bit about the feedback feedback loop for that. Um, to really squeeze the best return on investment, right, of your bugs and and just overall having a bounty program, um, it's it's really recommended, um, it's my my opinion, uh, to integrate tooling enhancements,

right, to your feedback loop. This can look like analyzing and fine-tuning SAS rules based on vulnerability trends. Um, postmortem like Ps like Garrett talked about and more. Um, additionally creating and publishing DAS templates for regression checking of like severe vulnerabilities which you know they'll they'll benefit your organization. Um, setting a criteria of what meets the bar for these enhancements will go a long way. Um there's a lot of ways really to like squeeze really what the ROI is and why you're really running a program which is something we'll we'll cover on the next few slides. But um another that means a lot to us is education and enablement right it can look like a lot

of things. Uh if your org has a security champions program right uh surfacing and having this like clear kind of like group with them right of like hey here's an interesting van and here's some case studies um that have come in in the last month. uh it could be really fruitful. You inspire those folks with like real world examples with like org context and they take it back to their own teams and and surface it between their teams. So I've seen that done and and it it's worked very well. Uh secondly, doing a bi-art quarterly or a yearly internal write up on your most interesting vulnerabilities um including your most complex or your most funkiest, right? or

the one that require like a seven vone chain to get like a rce. Um even those are are like the most impactful, right? Um and they could be very helpful in highlighting the quality of research and uh methodologies that external audiences uh provide to your program and and assets and that that's like one way of seeing you know the value that your program is is bringing you. Um, another which I feel very strongly about is highlighting vulnerability trends um, and like the actual attack and discovery techniques um, to other teams in your organizations may interest them to like kickstart internal research and plan companywide initiatives and to tackle and improve the security hygiene of your

organization. Um, we'll go on to the next slide. So, how do you staff a team that you know a structure for this? Uh, and the answer is, which we all hate, is it depends. It depends on your assets. It depends on, you know, how big you are, how much money your company has for for this. Um, I've se we're going to talk a little bit how we've seen it split up in the industry. We've seen PER and bounty teams as completely separate teams and there's like a dedicated bounty team, a dedicated PER team. Um, you I've seen some some bounty teams where they are the first level triage. There's like three or four of them

depending on the report count. um and they do validation. They do, you know, um the handoff to PERT as like the coordinators for for that work. And then Gary will talk about what he's seen. >> So I've worked at a place that had first a PERT and no bounty program. The next place had a bounty program and no PERT and finally where I'm at um managing both the PERT and the bounty program. And they're pretty uh inseparable. They're they're really the same. And so with our PERT, we're doing in-house uh triage. you know, we're uh not just like sending off reports to developers to say, "Hey, is this an issue?" Because, you know, we really want to do our own

due diligence. Um and then for those things that kind of based on internal criteria we feel need extra attention, you know, they kind of spill out of the PERT or out of the bounty program, um that's where the PERD starts coordinating with like incident commanders and other parts of the organization where um for some reason, you know, a a given report or you know, a certain issue we feel needs some some extra attention. So, I'll ask Jeff the big question, which is, do you do on call? >> Hold the tomatoes. Hold the tomatoes. Um, it's it's a big question. Uh, I think, you know, what I've seen and as I've talked to people, um, it really

depends on has it been, you know, discussed with your employees? You know, is it clear going into the job that there was going to be expectations of some sort of on call in the future? Um if it wasn't and you joined and you're like nope we don't have a on call uh we just try our best that's it. Um and then all of a sudden it's like we're all on call now and you have to have a page of duty. You have to have a separate phone. Like that's a huge change in culture and expectations. Like that's tough, right? And so that mindset and that shift really needs to be thought well crafted. Like why do we need that? Why do we need

an on call? Can we tackle that by hiring different time zones which is something I'm seeing more often. um can we satisfy a 24/7 coverage by doing a you know um specific geoloccational um hiring? Um so really it depends and you know we all hate that answer but uh that's kind of what I'm seeing. Do you think we should do call? Uh I don't know. So what we did was we kind of actually naturally got there mostly. So we're in four different countries, you know, a number of different continents. And so we kind of ended up that way anyway because the company has um offices all over the world, right? And so it wasn't really intentional, but it

has been helpful. You know, weekends can be a little sketchy, but um no, in in theory, no. In practice, kind of. >> Agreed. So let's talk a little bit about disclosure. Um, when it comes to disclosure, you really need to have a repeatable, durable process on on where your stance is with disclosure and what you're going to do every time, right? Like every experience with every different disclosure request should realistically be the same. That experience should be trans transferable, right? whether it's a PhD group from a a university or if it's a you know researcher via whatever bug bounty platform you are in um you need your stakeholders to have you know a say every time you need to have a specific

playbook for it uh you don't want to do it on the fly every time uh so you need to manage legal and comm's um expectations you need to have a process on you know SLA on when you get back to the the reporter um and and what your expectations what are their expectations ations. It just needs to be repeatable and durable um in my opinion. >> Cool. All right. So, buckle up. We're going to talk about metrics. And the biggest reason to go down this road, um I think a lot of managers feel compelled to just have metrics like what are the numbers without thinking about why you're really doing it. So, for me, when

we were asked to as we matured our team over the number of years, we got established, but then stage two is like let's become efficient about it. So, we were getting pressed for metrics and I had to sit down and think about like why bother sort of thing. And my biggest thought was let's have a good way to tell a story to justify this program. And that probably applies to you as well. Whether you want to start a program or keep yours going or expand it greatly like we did when I said, "Hey, I want we've done this much over the years. I want to go three times as large next year." Really had to have a solid

case. So, you want to be able to answer a question like, is this program worth a million dollars plus a year to your business? Um, what kind of like provable results are are you going to get out of it? So, to put myself on the hook, we went ahead and established an OKR, a objective and key results, uh, but one that was at the CISO level and therefore board visible. And it had it was tracking an aspect of the program's uh, health that was really close to its impact. And so it kept us on us because every quarter we had to go back and be like, "All right, are we on track with this?" You know, I'm still employed. We

hit it. We were green. Little sketchy there in in the uh in the middle, but um that's one way to to have some external accountability on the dashboard along with all the other CES priorities where they can check at a moment and be like, "Hey, is this going well or not?" And then qualitatively we try to tell a story about how the program has like expanding values. So one is if we're getting in a number of different ways a variety of reports and so variety could be man these different applications each have findings that feels like a lot of good code coverage. Um another is if there's like a variety of all classes it's not just all

cross-ite scripting you're like hey we have some talented people who know about more than just one or two bug classes like it's pretty solid researcher pool. um severity reports like I'm really interested in those like truly critical uh high impact reports a lot more than you know just like a deluge of of low low impact stuff still valid but you know less important and then we do some trend analysis where again you know at some point uh your execs come around they're like hey awesome program but like why do we still have reports so like what are some of the deeper root causes of some of these symptoms and sharing that information back to the

execs and the of the uh product security department that is involved in in the life cycle. >> Sweet. Let's talk about the program itself. Um to start with researcher retention. Um we are competing with every other program, right, for researcher talent. Um and researcher talent can leave your program and go jump to another one immediately like pretty quickly. So the experience needs to be good. The experience needs to be repeatable. And you need to measure are people coming back? Are people staying right? Um are you seeing a steady flow of reports from the researchers that that you really like and that you really enjoy working with? Um at the same time, you also need to manage that by focusing

on researcher growth. Um like keep the ones that you like, keep the ones that are doing well, keep the ones that are giving you a good experience and you're giving them a good experience, right? Um and how are you growing that? um you could do that in a lot of ways and doesn't just involve like a campaign that means like paying may you know way more for a month and then go back to your normal bounty reward pool right um some ways that I've I've done it without you know focusing on the money aspect which is great right like I'm not saying don't pay a lot like you should be paying a lot um as much as you can uh

but some non-monetary ways you can really do it is um like cyber security awareness month hey like we're going to write some blog posts about our coolest researchers, right? That's free, right? Um, you reward researchers for their time uh, writing that. You interview them. You give them like a public blog post in your website and highlight, hey, this person found this really cool bug. Um, their specialties are XYZ. And um, and yeah, you you give them something they can point to and like, you know, you also monetarily award them. But that another thing is like bug bounty at your company is like specific swag. So, like something I did recently was gify swag at at GitHub and we launched a swag

store that along with your monetary awards, you would rack up points to get like bounty exclusive swag for your your store. Um, and you know that the when you see someone walking around, you know that they got that from from that program because they were, you know, submitted like an RCE. It's kind of cool. It's like pretty cool to flex that. Um, and so we saw researchers really like that. Um, and I could go on, like there's a lot of non-monetary kind of perks that you can provide, and they're not that expensive. Standing up a swag store was less than $50,000 for like 2 years. Um, and that includes like shipping and international fees and

everything. So, it's really not not that much. Um, and I really went off script, but um, but those are things that I really love. Um, and another thing too is like life hacking events or just like like campaigns, right? like not everything has to be a giant event, but just a focused group of like, hey, 10 top 10 researchers on our program. We have this beta feature that we really want to get like stress tests before we go public. Um, you have two weeks and you know, we're going to inflate the bounty tables and you'll have direct access to developers. Um, and here are like the bonuses and here are the things we really care

about. um like we don't think it's possible to break out of this environment. If you do it like we'll give you a sign a blank check um and that that challenge alone is is pretty sick, right? I'm not saying do that, but um but like really focus on the things that you're like I don't think this is possible and see if people can do it with like a very special group of like you know 10 20 people and spin up like a communication channel just with them. Things like that really focus on um and and on like researcher growth and people really like that. Um yeah I think you have >> cool and finally we want to focus on the

team uh that is you know helping to to drive all this and you can get part of the picture of your team's health through metrics. Um it's not going to tell you everything but you want to know how your team's doing. And for instance one metric you could uh take a look at is time to triage and see if that's going up or down. Now, that has potentially a lot of factors going into it at once. It may be a little difficult to try to figure out what what exactly is happening. But, of course, if your program is growing like crazy, you've got a ton of volume. I wouldn't be surprised if your time to triage goes

down. Like, people are busy. They're probably tired. I get it. But otherwise, there could be some things in that metric that help you identify things that people that your folks don't want to volunteer to you. So there could be some knowledge gaps in there on uh the particular product especially if you're in the business of like buying companies and you're like hey what is this brand new thing uh we bought you know we don't have any training on it. Um these numbers may help reveal opportunities where someone can uh scale up a bit and that's where you get to do some of your individual goal planning map a training goal to it some training budget. If you

see like the majority of the teams like not familiar with a company that just got acquired and everyone you know probably needs some of that that information sharing and then there's also opportunities to just look at the efficiency of the process and be like hey there's some tooling gaps here and then realizing we're not necessarily our needs are not just our own but some of these tooling gaps actually probably help the wider company. So if you're able to knock that out of the park, make some of those tools, then you get to benefit other parts of the organization as well. Um especially as your company's growing where you have like hundreds of not developers but

developer teams and like thousands of of developers. >> So let's keep talking about researcher cultivation. Um re researchers can be strategic partners. they have a direct and niche and particular set of skills that can enhance your organization's testing and processes. Um, you know, you have all these teams that are good at what they do, but sometimes researchers usually they offer something super special that like the teams that you have existing, the tooling you have existing already, um, don't. And so feel free to give them access to beta features. It's not scary. It's not as scary as you think. Um, if if it's beta and there's a few customers already using it, a few folks already in it,

it's probably okay for them to to hammer away at it. It's probably better, too. You identify these bugs before they get to the, you know, to to the general audience. You could fix them and and and really like have that relationship with with the researchers. Um, one of my favorite things is uh proposing patch snippets to researchers um and be like, "Hey, you found this really cool bug. We're thinking about fixing it this way. here's like the code snippet for you know um how we want to approach it. What do you think? Um can you find any bypasses for that? Uh and obviously you have to you know compensate for that extra research extra time but um those

remediation ideas and those like recommendations are are really cool to get from you know the people who are directly finding these the these bugs. Um you'd be surprised by the amount of times that researchers go above and beyond um when given the opportunity to collaborate further on a report. um they love what they do. They love the, you know, they have a pride in the bugs that they send over. So, uh if you're, you know, if you're running a really fair program, they're going to want to engage further as much as possible. Um, one thing that Garrett and I talked a lot about when we were thinking about doing this was recruiting um and like

converting researchers to join your company, right? Um he he has some cool stuff on on that that he'll follow up, but um it's like a cool selection for the pool, right? as as people kind of get really familiar with your your products and your your tooling and just internal processes as you collaborate with them a lot. Um, and yeah, there's there's a lot you can do with with recruitment. Awesome. I love using our researcher pool as a talent pool when we have a position open up on the team. Um, we've done this twice over the years with great success. We probably could have done it more, but sometimes when I talk to researchers, they're like, "Hey, I like my lifestyle.

it's flexible. I don't want to do your like 9 to5 thing. So, um I was like, "All right, you know, you you got a good thing going." But I really like our talent pool because there's very low risk in some and technical ways to hiring this person. You know, if you have a difficult proprietary um custom product that takes a long time to really get to know uh the the inner workings of it, someone who's demonstrating very advanced findings against that platform, I'm like, "This is great. They're going to be awesome on day one. um they won't even see this as like having much ramp up time or or learning curve to it. So, at least as far as like

the technical skills of um you know using some of your proven researchers, that's awesome. You may have a good indication of like their temperament as well. Like have they been pretty patient with your team? Have they not lost it on you? Like pretty solid, pretty solid choice. So, I would absolutely do it again. And some of the best researchers have been folks who are working at uh implementation companies, contracting companies, putting the software in place for um pretty complex customers. So, ironically though, if you are reporting on the wild success of your bounty program and you're like, "Look at all these high impact findings. This is amazing." Um, and you bring some of those folks in house, you're kind of

undermining the success of your own program, though, because then you'll see the volume temporarily drop where you're like, "All right, well, we we brought these folks in house. It hurts anyone who's really wedded to the bounty metrics, but I think for, you know, the greater good of uh of the organization, you know, it it just makes sense. >> Cool. Just want to quickly touch on alternative rewards. We didn't put it up there, but um we talked about swag earlier, right? Another thing is Hall of Fame. Some people use it. Um it's been around for a while while it's just a way to acknowledge somebody on your website. Um, I think it's cool, but it should be

partnered with rewards and all the many other things. Um, one of my favorite though is uh co-authoring content. Uh, hey, someone found this really cool bug. Write a article about it together and put it up. I know your PR and legal team might be a little scared hearing that, but uh, if you make it PR and legal friendly, it might be okay. Um, but sometimes the bugs are so interesting and and unique that it's worth a public blog post and and a partnership with the researcher that submitted it. Um, so, so I would say we definitely explore that that that avenue. So, something that Garrett and I talked about a lot last year, um, and a lot of

other program managers is, uh, do you need to go public? Is is going, you know, is having a p private program bad? Like, have you've had it for three years? Is it bad that you don't go to p go public on the fourth year? Um how do you measure you know uh success from going private to public? Um and why like justifying that that is is important. Um how do you measure the scope right for like why your program is being successful as a private program? Why it might not be successful immediately for a public program? We have some advice that we want to go over on that because it's it's a really touchy subject.

>> Yeah. Yeah, it all started when I talked to Jeff and I was like, hey, if you were to try to make a measurement of program maturity, must you be public to be considered a mature program? Um, and so, uh, separate from that, I'll soapbox a bit. You know, my my background is prod. It's not enterprise security or anything. And not all bounty programs are enterprise programs. Meaning, some people some organizations have bounty programs to say, hey, tell us about problems in our whole network, right? not at all what we do. So our program is a a product program and where you run into like some conflict on expectations is your asset scope in a product program

may be like one line. It might be like hey this subdomain and someone who doesn't understand it uh might be like hey that's a very small scope I don't really get it but they don't realize is that one that you know one domain or subdomain hosting a product that is expanding exponentially on the back end you know tons of new features uh components applications and then behind the scenes backend services that do like data analysis whatever your backend services are doing. So that's to me one of the biggest things between like an enterprise program that's like hey look at you know we have a thousand subdomains and like all these IPs in in scope um versus a product program. It

may literally just be like one domain maybe a couple of the mobile apps but they're absolutely massive. And so one of the things we've been doing to try to communicate changes to our researcher base is we're like hey check out the release notes. Um if you know Service Now's release style every six months huge family release they call it. And so we're like, "Hey, here's all the new stuff involved in this uh family release. Go check it out." Right? So that way people know that your your software and your program is not stagnant even though you haven't added to like the asset table in a while. So measuring maturity for for your program, whether private or public, is

is interesting, right? Like are all the assets that you care about in scope um and if they're not when how can you get there, right? Um, is adding researcher base increasing the report volume? Is it increasing like the validity too? Um, are they valid or, you know, are how fast can your team respond to highs or criticals um from when they come in? And I think that's like the one that that's probably the hill I I would die on when it comes to bug bounty programs. Um, you really need to be able to respond to a high, to a crit, to a low, repeatable, in the same fashion with the same process, right? at this at a reasonable

time like and and you set that internal SLA, that internal reasonable time, but you need to be able to do it consistently. And if you start to dip, you should probably figure out what's dipping, like what's causing the dip. Um, are you having issues uh coming up with rewards? Cuz they're starting to get really complicated. Do you need to simplify your bounty table? Um, are they privacy issues? Are you seeing like a big influx in hey, I'm able to access stuff that you know that I'm not supposed to, but that's not in scope. Um, are you seeing like vols that you've never gotten before? You should probably revisit your scope and and your tables. Um,

>> cool. And that's it. Thanks for sticking around. We're just going to quickly wrap it up. Um, for bounty programs, uh, my personal, this is my opinion, but you should really start small. There's no reason to start big. Start as small as you can and really stress test your process. Make sure that every time you can do the same thing. You can triage a report the same way. You can find a owner for everything that you have in scope. Um and if that's changing like how how quickly can you find out who the owner is now, right? Reorgs happen all the time everywhere. Um so like how fast can you track a change in you know your

own internal catalog or whatever whatever you call your ownership. Um so stress test as much as you can. you make a change, right? Like a whole process change, stress test as you go before you, you know, kind of like start using that publicly. Um, and automation, um, you'd be surprised at how much it just using automation in terms of integrations and a custom app. Um, you'd be surprised how quickly and easy it is to create that and how much that helps you and like that really raises your efficiency. Um, I've seen it at three companies now and it's it's it's amazing. Um, for bounty programs, uh, resources, uh, that we love ourselves are the bug bountyci.org.

Uh, we're pretty active in there and there's a lot of stuff that we put out for other program managers that are kind of like tackling the same problems we are um, in the bug bounty space. And you want to talk about PERT? >> Yeah, for PERTS, again, if you are tasked with the daunting responsibility of going from zero to you need a PERT, uh, you can start with the first.org or PER services framework. At a minimum, it gives you something to point to to say, hey, you know, here's the collective judgment of a number of experts in the field, you know, like we're following a road map. I'm not just making it up as we go. So, that's been really awesome.

And then it gives you a little bit of a challenge. You know, once you hit level one, um if you're competitive, then you're like, what's it take to get to level two? And then level three, um TBD if there's going to be a level four. So, I really enjoyed it and it definitely gave me, you know, something to work with right right when I needed to start. So, thank you so much. We got seven and a half, eight minutes for questions. >> All right, let's give it uh Jeff and Garrett a round of applause. Great job, guys. Um, please submit questions via Slido and they're going to pop up on screen for me to read here over the mic

so they can be captured um and streamed. You can submit your questions at slido.com or slyd.go, excuse me. Um, and then enter the code besides SF2026. That's besides SF2026. And then select theater.

be kind of funny to send questions to another theater though. >> Yeah. >> Yeah. Don't don't send questions to another theater, please. >> And you can also vote on your favorite questions.

>> Yes, that is uh sli.o B O for the URL and then the code is bsides SF 2026 and we are in theater 13. So select that one >> I think so >> talk about >> durable process. Yeah, >> stay hog. Can't do it the same way. >> All right, I am seeing a few questions start to populate here. Uh, from anonymous, can you speak a little more about the KPI/RS uh specific good or bad metric examples? >> Oh man. Uh, there's a whole book. Uh, so and I think it's a whole art science. So, OKR objective and key results, the theory being the objective is where you want to get to. An example, if you read

the well, I'll throw out another takeaway. There's a book, measure what matters. Um, and that's going to tell you how OKRs work. And the idea is like an objective is like, hey, I want to win a race. And the key results being a few bulleted things, maybe like three to five. And the theory is that you write out this list of like three to five things. if you're able to accomplish all of them, you will necessarily have met the objective, right? So that's why it's like, hey, these measurable key results are important because collectively they're going to help me uh meet my objective. So it's tempting if you're running a program just to be like, hey, let's have

a graph of report volume. Super easy. Um but in my mind I'm always I've got the challenge of what's what I feel is good to you know communicate value of something and what's measurable don't always intersect and that drives me crazy. Um, but instead, for instance, I'm like flirting around with the idea of like a report ratio of just more and more criticals, less and less like uh low severity things, right? To try to say like we're just getting extremely efficient in seeing that, you know, most impactful report volume. >> Um, real quick, I like to drill deep into like the the trends themselves. Like if we got seven crossite scripting in the same week and they're pretty much

all related or they have a root cause, that's like a really good metric to bring up to not just leadership but also other teams like we talked about earlier. Um the so so I'd say like that's an example of a good one like focusing on classes themselves correlated with with scope. Um yeah, >> Jackie asks, "How is this space changing with vendors like Expo increasing the velocity of submissions and replacing human researchers >> gets to be burdensome, but I mean if you're hiring tier one in between you and the researchers, it could be their problem. Um but I am seeing that where like early in the like kind of that chain of phases yeah there's there's an

increase in volume. I am wondering though if it's temporary meaning like when I was talking about old vulnerabilities that are getting burned down um are we just going to see this huge influx of things that have been around for a few years and then it kind of dwindles off assuming your organization's not like introducing tons of new vulnerabilities every day. >> I think it's really new. Like I generally don't have an answer. It's like that new that like influx of ch long trend of reports that are coming in. Um so so I don't really know. >> Latin poppyc asks can you speak to how you broaden your bug bounty program across borders or handle global

researchers? >> Yeah. Make make like sanctions the vendor's problem and >> take care of them. One of the awesome reasons to have a vendor in between you and the researcher community is they get to worry about like payments, taxes, um identity verification, sanctions, like it's it's pretty sweet. Um and it helps you have like a platform for intaking uh you know reports and whatnot. So we didn't have I really don't even often look or could know where someone is based actually unless they specify it in their profile. I don't think I know you know where someone's based. >> Yeah. Yeah. vendors usually take care of that and that makes >> swag got interesting when I was like man

we are shipping t-shirts all over the world >> yeah so I hope that answers your question I don't remember the name of the person who asked that but >> anonymous asks how do you know the right amount of bounty to pay >> it's an arms race so you were going to mention actually you do like a calibration every >> yeah I that's one of the bullet points I missed to to mention um it's I I usually recommend every 6 months at least to look at your surrounding programs. They don't really have to be in the same industry, but just in general looking at other programs and what they're doing well. Um looking at their bounty tables

and seeing if you're way off in a good way or in a in a bad way. Um like if you're paying, you know, $1,000 for a crit, you're receiving a lot of crits, like you should probably bump that. Um or you should recalibrate and figure out why um and what that decision was. So, um, so it's hard, but I I I think there's there's a lot of value in seeing both in your industry, right, where your company is, and outside of your industry, like what the average payouts are, and and just getting direct feedback from researchers, too, is not is not off the table. Uh, you could straight up ask them like, "Hey, are our

rewards fair?" Um, and and you'd be surprised at like the honest answers. One quick thing on that one is our program started like 2015 and we were doing our calibrations and I looked at the footnote at one point and was like oh our cohort changed like we had hit like Fortune 500 status and so when I went to our vendor and I was like hey I want to calibrate but this time like who you know who's our new set of competition right when it comes to bounty programs. So all of a sudden my numbers were different because you know we're comparing ourselves to other you know much much larger organizations. All right, and that is all the time we

have for questions. So, thank you so much everyone. Another round of applause for Jeff and Garrett.