Life of a Bug: An Insight on the GitHub Bounty Program

all right everyone for those of you who just entered this is theater 15 the best audience in b-sides and I want everyone to give a round of applause for Caitlyn and Jeffrey they'll be our speakers

thank you welcome everyone today we're going to be talking about life of a bug and insight on github's bug Bounty program a brief agenda what we're going to talk about today so we will both introduce ourselves then Jeff is going to give a high level overview of our Bounty program then I will jump into a high level overview of our P cert program we will then introduce a mock bug which we will uh walk you through the process of our two teams collaborating on the bug from triage through the whole IR process Jeff will go over some tips and takeaways and then we'll do any q a so my name is Caitlin I am a product

security engineer with GitHub I lead our piecer engineering team which is a subset of P cert where we dive into more data-driven metrics of our investigations digging into variant analysis Trend tracking really diving into the pain points of what we're seeing across pcert my background is in corporate security moved into product security and incident response when I came on board with GitHub about two years ago hey everybody I'm Jeff several years in phone management slash you know managing triaging multiple bug Bounty programs May senior project engineer at GitHub tons of bug Bounty stuff that we do there huge researcher engagement Advocate will go down some of the what that looks like soon um and just passionate about all things

security let's talk a little bit about a high level of our program and what bug Bounty looks like at GitHub we manage two programs a public program that has a very extensive scope which researchers love and we love that they love that a private program 25 hackedo cats this is something we've been operating for like two years three three years and slowly we've been revamping it and offering more beta features and cool engagements for our researchers who really know our products well and we have a dedicated GitHub staff this is something that's kind of unique we have a team that solely uh focuses on our program um yeah I'm part of that team researcher committee engagement one of the my

favorite parts of working at GitHub um we recently launched a public Bounty swag store might not seem like a law but we got a lot of feedback at Defcon we met up with some hackers and we're like hey we'd love to kind of rep your your your stuff you know your your swag but more specifically Bounty specific swag you know and so we launched a store that's just for our researchers very funky very cool and we love it we do a lot of blog posts where we highlight our researchers and we just kind of give them a microphone and let people know why they're hacking on our programs some of their favorite bugs that they found

Etc um yeah we offer exclusive perks cool beta access which everybody likes right hunting on a new feature is always fun and large rewards we award up to 50K which is you know a good amount in the industry right now uh we've been operating for about nine years we've awarded about 3.9 million so far in Bounty rewards and uh we've hit 1.5 million in 2020 2022 which is great right uh we have top researchers testing that keep coming back and I think that's that's a testament of our program uh tons of stuff around Community engagement like I said earlier and uh that's why we focus heavily on it and it's this program the way we have it

structured really strengthens our customer engineering and IR partnership and its own unique ways and we'll highlight how that what that looks like throughout the presentation um here's her here's our scope breakdown just a quick high level almost everything is in scope almost everything is eligible for rewards um Dependable code space is the types of products that everybody uses on daily um you can check out our scope and our own personal site on ballion.gettable.com scope or our policy page on hacker1 what's not eligible is basically just open source projects we welcome contributions there but most of them are out of scope and known vulnerable software we don't really pay for those uh Upstream dependencies we sometimes help coordinate that but it's

not something we can usually award for and a few more things and you can check that out in our ineligible site so pisert from a high level we are responsible for the product security vulnerabilities that impact GitHub products and our customers you may be familiar with more cert uh our team is called Thor on the other side they are the corporate security side but of course we're doing product security within psert so we handle any security reports in three different ways first we will start with a lead so when a security report comes in uh we will always spin up a lead with NP cert and this is very important part of the process where we can start generating

those metrics that we will utilize throughout this whole process and then once we have that lead validated we will turn it into an investigation if a investigation we deem is more severe or involves data that may need a specific notification requirements we'll turn that into an incident so you may be wondering where are these security reports coming from well thank thankfully we do have a very secure minded uh internal group of hubbers who will often report security findings directly to us and assert assert slack channel that everyone is uh I can access and is open to but we also get a very large number number of external reports right so a lot of them come through our Bounty

program but we also do monitor blogs Twitter uh security companies will write into us we do get third-party reports occasionally and of course our customers and users as well So within P sir it's we do receive a large number of reports and one of the most important things about a program that is very process driven and receives so much coming in at once is automation so we are very very big on automation uh within pcert at GitHub and one of the things we utilize quite a bit are our chat Ops everything is integrated with the GitHub platform and slack so that we are essentially just triggering a bunch of chat Ops and dog fooding our product

which is something we do a lot and uh it's really good about taking out a lot of the third party products and not needing them and just utilizing our own tools so we do this with leads I'm going to walk you through how we do that once we do introduce the bug and then how we go through that whole process with investigations another thing we do with automation is something called slash commands this is something built into GitHub and we use this for prioritization because we get so many reports in we need to know what to work on first so we use these slash commands and I'll show you this as well to determine this prioritization we will

slap a label on him for ACT attend track and based on those labels will determine what gets acted on first cool so let's reveal bug from the beginning to the end of it uh it'll give an Insight on the Bounty side of things on the piece or side of things and how they are intertwined um so in this case we use hacker one to you know to receive these bugs in this case we got an idor on Mona Hub um and yeah it's a fake bug if the red didn't indicate it it's definitely very fake um that I created and so this is what happens when the bug comes in in this case uh that's that's me right so I

triage the the bug I reproduce it I'm like oh this is valid so we have automation ourselves to make this faster right um and in this case the report ID is that number there and we'll run a command that says hey go this is valid let's go ahead and triage this to the engineering team that owns this we'll throw it over to the repo and this bot will safely throw everything over to the repo and that's when the person who's triaging the the report will say hey here's my reproduction steps on it here's where I think this is happening um I was able to maybe escalate this um and in this case you know I

charge it over I added my summary which I didn't show but it's down below and then I realized that it does impact uh GitHub Enterprise server which is ghes you'll see that throughout the presentation and uh it's about a medium plus so in that case we're gonna have to Loop in piecer um we'll talk about that so at this point bug Bounty is going to Loop into Loop and piece art and we will get our on-call person or whoever is available to get started on this lead so we utilize this chat up that we have built in-house uh within slack and it will uh ask us to answer a few questions you can see in the screenshot we'll assign a

psert IC added title add the summary assign it to the team because our counterpart Thor cert does this uses the slack chat up as well and then of course we're plugging in the type this is coming from Bounty so that we're getting these metrics going within the lead immediately as you can see in this screenshot as well it does put the lead directly in a GitHub repo and this is where once again we're utilizing the GitHub product to start tracking a paper trail of our incidents so we always have a reference back to these so once we have this lead spun up and an issue created we're going to go back to the initial reporter and we're going to

acknowledge the fact that we got this lead and we've started the um process of tracking it and the next step uh will be to determine that prioritization and of course impact so once again we're going to go back to the Automation and use the slash commands it's going to ask us a series of questions and it's going to spit out an action and based on that action you can see here it tells us to act but not only do we want to rely on this automation we also have to really think about what this bug is and think about the impact because the impact to the GitHub customers and users and our product is really the most important thing so we

know off uh what this bug is in idor that the impact is an attacker could manipulate user data not good so we know at this stage that we need to convert the lead so once again we're going to go back to our chat ups and it's going to ask us a series of very very well known incident response questions like severity sensitivity so we're going to assign a TLP and then it's going to ask us some more slack related questions do we want to make private slack channels that's going to be based mostly on that TLP so this stage we're converting this lead directly into an investigation our chat op is going to do this on the

GitHub issue side as well so it's fully documented in the issue we don't have to go back into it do anything manually at this point it's fully converted and one of the things as Jeff mentioned is that we know that because this impacts gags gonna probably GitHub Enterprise server we will need to issue a CBE so that's another reason why we're going to convert this into an investigation make sure that's fully documented make sure the pcert IC is responsible for assigning that cve getting all these details in there and getting it in the release notes for ghs so that customers know the security impact of a patch so at this stage we are beginning the

investigation so we have the severity set TLP of course our slack channels and our summary is getting developed everything is uh automated to this point and now the psert IC is going to come in develop that summary although I have a feeling we could start using copilot for this a little bit uh to generate generate that summary and we will take that summary put it in the slack Channel make sure it's pinned we'll have next steps assign Engineers to those next steps so that anybody who comes into that slab Channel knows exactly what's going on what's going to happen next and we don't have to answer questions about any of that so simultaneously while this

investigation is getting spun up we're also working in the response phase so this stage we are dealing with the most important priority and that is to stop the bleeding so the good thing about working with our bug Bounty Partners is that they will often engage engineering before it ever even comes to pisert they will get the engineers engaged on this bug a mitigation will often get being worked on very very quickly sometimes before it even comes to pisert which is awesome for us we love to see that if not we will definitely jump on it and get engaged with engineering team who owns the issue uh we'll ask them for a root cause they'll develop that

mitigation and then they will also develop those backboards for gags since we know that DHS is impacted while that's happening on the engineering side the pcert IC will also be determining scope of impact so they'll do that by reviewing logs looking for evidence of exploitation all this is going into the slack Channel or issue for full documentation for review and determining our next phase and that is a notification so with every incident we always ask this question uh our notifications needed we try to be as transparent as possible with GitHub so notifications are a big part of the job we do for this bug we know specifically that a cve is necessary so psert IC is

going to issue that CV work with the ghes team to get that in the release notes for the next patch make sure that all parties are aware of this specific bug that's not the only notification that you may have seen from GitHub so we also have github.blog which is our blog post we use a security tag for any security news that we have that goes out we've had a a good number of blog posts trying to be as transparent as possible about the bugs that we found on our platform and then we also utilize our change log for bugs that may be smaller but we want users to know that a change has been implemented in.com especially

if it may impact them and then of course we also have direct emails if we're able to scope for impact and have a list of users then we will send out direct emails this stage bug Bounty is going to work with the researcher all right so at this point we're backboarding stuff right and and we're getting ready to wrap stuff up from the piece search side so Bounty is also starting to wrap things up by asking a researcher hey this qualifies for credit uh cve credit which is awesome right and our site states that um most most bugs with GTS there's like a specific language but ghes will usually get cves right for a specific severity

maybe and so in this case we asked researcher hey not shown but in the background we're doing constant communication Ryan so we'll tell them how would you like to be credited for this if you would like credit at all and at the same time we're also doing in the background right we're getting together and we're deciding what the payout should be which is a very transparent consistent process the Bounty person who had it from the beginning will pitch and suggest a payout based on our external um and public uh severity um payout guidelines and in this case you know we suggested 10K and that's because the this is you know what it falls under um based on our external site and this

is what we've paid out in the past for this type of idor um and then we also you know talk about the impact and all that so uh every week we have a payout call where the whole team will get together and review these and all of us have to agree on it and if we don't then we go back to the drawing board and we decide you know what's what's the fair amount if we missed the mark on the first round uh which is cool from my side like when we go and pay out the reporter uh you know a researcher they'll feel like it's fair and you know we usually don't get a lot of um like

back and forth on that and it's because it lines up directly with her with our site um and yeah for more information on our rewards our Bounty side is extremely up to date so now we're working into the post investigation side of the house here uh and that is going to be the pcert IC is going to take all this information everything we've done throughout the investigation and put that in a blameless post-mortem a GitHub we're very big on promoting a blameless culture it's all a learning experience for us and um it's very important that we uh maintain that so in our postmortem we're going to include of course a summary timeline RCA Lessons Learned

after actions uh because we don't want to hold up the investigation with after actions or any lessons learned that we want to carry forward across the org so we're going to list those there put them in pirs post incident repair items and then close out that post-mortem this is something I know I mentioned co-pilot earlier but this is something that we do uh use Code Pilot for sometimes it does help us and any help we can get in getting these reports out faster it's much better so this is where my team comes in piecer engineering we're going to work through prevention prevention is a big thing right we want to reduce the number of incidents that

we're seeing come in so we do things like bone review meetings where we review our postmortems and we share our findings with the greater org we do varying analysis where we really dig into uh bugs and we try to find variants in different parts of the code base we work with our code analysis team to develop detections and Sentinel and code ql queries and then of course all these become posts incident repair items that we're tracking elsewhere cool so like I said earlier the you know payout was suggested at this point we're going to award the payout um and you know this is one of the languages that we use when we uh when we

pay out the reports so wrapping stuff up um we're giving a coupon free of charge get a pro for life um and then in this case it because it got a cve uh this is something we've piloted maybe a little more than six months ago where we started to try to share with the hacker Community besides you know what we're up to actual summaries of like the reports that we're getting um and I know there's a lot of work to be done there right full disclosure is what everybody wants um and this is you know one of the first steps there and so in this case we tell them hey we tell them hey there's a CD that's being

published we would like to disclose a summary and you know some of the report contents and so we're excited to share with the hiker Community if you if you're okay with that and this is kind of like the process that you see when it happens so let's talk about some tips and takeaways by the way there's a lot more we could cover you know but 25 minutes is 25 minutes um so transparency and trust is something that hopefully was obvious throughout the the process for both teams and you know engaging with the researcher with the with the engineering teams um and researcher engagement is hard right because we want people to to keep coming back um and everybody has their

own story their own reasons for bug hunting and so we try to have a very consistent and transparent process when we work with researchers and something like that looks like is like having consistent communication um so we've partnered with like legal PR to make sure that when we communicate with folks it's consistent and the language is usually uh the same you know with all these and along with that customers and users are just as important and so they hear things from us first just like uh Caitlyn kind of went through some of our notifications um and there's a lot of value in being transparent like that and I we we've seen uh a lot of good things come from

that another thing is maturing automation is important that's very easy to say right but it is something that we put in practice as you saw bounty has their own automation it makes things very easy very quick Treehouse things over you know all the report content goes with it that's awesome um it'd be very hard if it was all manual right which happens in when when you first launch a program I'm sure um but yeah same thing with piecer and so automating the easy stuff first is is the way right and then you build off of that based on the pain points that you see in uh the you know the the time that you have to to address those

um and then lastly Bounty and peace your teams for us right are extremely closely aligned we're under the same uh director and so what that causes is a very quick and seamless process for incident response so when the bug comes in you know we have these processes that make sure that we can go from receiving it to fixing it extremely quickly because the teams are so close and we have built these processes together um and it works it works amazing for us and that's kind of that's all we have for today um again 25 minutes is is all we had but there's a lot that we'd like to share more uh if you have any questions we

still have about four minutes for questions um and you can follow us on Twitter and or see us in the lobby to to chat further thank you theater 15 round of applause the best audience at b-sides folk who have questions please raise your hand slido is down so we're gonna do a microphone if you guys want to project save with your chest go for it oh I'm supposed to say good okay uh sorry you guys can pick

yeah oh for pizza

um and working with the engineering team you know sometimes they do last longer we do have yeah so yeah but I'll say our engineering but yeah are we doing microphone or any other questions

test test so I really like that your scope is pretty wide uh how was that decision made is this like a top-down thing saying we want to be super secure it's bottoms up saying let's find out and how did you get the buy-in I guess from the product teams yeah yeah that's that's a great question it varies right based on uh uh you know like you said right uh leadership and some leadership could get nervous right with like hey this feature we don't really want to advertise that's in scope or whatever right like for us it's uh because we have such a blameless process it's uh everyone's excited to to launch a product and then put it in

scope to get additional testing because we have all these different testings in place right we have a bunch of teams that run their own scanning tools their own testing and everything but we see Bounty as like the seventh or the eighth right it's not the one we rely on and it's not the one we get the most bugs with but it's the the nice to have you know extra testing and the ones that aren't in that public scope they're most likely in our private scope you know with with our private bug Bounty researchers so we're getting eyes on these all the time and we're seeing a lot of value in that our customers are loving the fact that we're okay with

testing our stuff and in great depth yeah but it's hard to get by and if if the culture doesn't follow that so you have to set it you know and then nurture it easier said than the any other questions okay yeah

I understood the question

um

yeah that's a great question I think that might be a little out of out of scope for us to give you like a valid answer on unfortunately um but if you're curious and if you write me on Twitter I'm happy to track down the right person to outside that might be closer to like assessment or you know some of the the teams that work closer with like when the products are launching like that or where they're at with launching those but yeah I can't I can't really help you with that fantastic questions folks again theater 15 Applause for our friends Caitlin and Jeff foreign