BSides PDX 2023 - Securing your Open Source Project (Jose Palafox)

[Music] my name is Jose Pala Fox and I used to live in Portland I'm really excited to be back here for this event so thanks so much for having me um I moved to Portland in 2007 uh when I got a really generous package from Reed College uh so finished up at Reed and uh helped start a software company here called Puppet Labs uh if anyone's familiar with with puppet uh we did really well for a couple years and then you know didn't didn't end out so great but you know it was a really fun ride and um you know Portland was my Portland was my first real home you know where I felt really comfortable so I'm just super happy to be back here so thank thanks so much for having me um I work at GitHub and my job is Loosely to work with all of the largest software companies in the world and talk to them about open source security um help them adopt uh better security practices and help them build more secure applic that we all all use so what I'm going to talk with you about today is really um kind of the state of GitHub and and maybe why you should care about it as a security professional um and and I'll take you through some of the things that we've been doing at GitHub over the last 5 years or so uh to really improve the quality of um security of Open Source packages on GitHub we've introduced a ton of tools and to share kind of what those things do how you can use them for free on GitHub if you you interact with GitHub um and then I've kind of got a Persona based approach to give you some actionable things that you can take away today kind of no matter who you are in the room um there's stuff that you can do to help us secure the open source software on GitHub so uh that's sort of the the structure structure of what we'll do today um so so what is GitHub who has a GitHub account everybody all right fantastic tech room we nailed it got it all right so you are one of the 100 million plus developers that are on the platform but it's also the home to most open-source software um and that's probably how people are first familiar with with GitHub I talk to students all the time and they'll say GitHub that's the thing that I download you know our scripts from and I really like that and like a yeah helped you pass your stats class um so you know it's it's great it's a place where people come and they share they share code um and and why you care I think about it is because open- Source really changed the way that Enterprise applications are built and you know anything that we interact with today whether it's you know the wristwatch that we have or the phone or the car we're driving in or the elevator we're riding in most of the software that runs all of those things is based on open source software which hopefully is a truism at this point this used to say open source is changing but I I've changed it to past tense because it just seemed too too obvious that um we use a ton of Open Source now and and all of that lives on on GitHub um a couple years ago we started having this problem on GitHub and and it alerted us you know to this responsibility that we have to help secure open source software people were leaking their API credentials all over the platform and this happens you know 300 times a day people will push a live GitHub token to github.com malicious actor Will scoop that up from our events API and then they'll take over that account so not only were we dealing with um a lot of account takeovers um you know it was starting to put some of the open source software on GitHub at risk because if a maintainer lost their own identity on GitHub you know somebody else could uh create a supply chain attack out of that you know identity of the maintainer that owns like a popular package uh so impersonating somebody on GitHub is a really bad thing and this was happening you know 300 times a day so we had a a serious problem so we built some tooling I'll talk about to address this but this was like the first big like Whoa We got to do something um you know this is more of a longitudinal thing that we looked at over time but what we saw over the years and from running static analysis tools on GitHub uh the number of security vulnerabilities that we found on the platform basically stayed consistent with the number of lines of code that were being written so even though people were becoming more aware of security vulnerabilities and you know better application uh coding practices we were still introducing loads and loads of vulnerabilities and that's what you see with this blue and red line basically staying in the same same curve here this is a complicated Double Y axis and one of the lines doesn't have a uh an axis so you know bear bear with me um I promise it's not false news um the purple line there is really you know people becoming more aware of it you see this sharp spike in uh 2018 2017 time frame where people you know started commenting a lot about how their PRS were resolving security vulnerabilities but still uh the number of vulnerabilities stayed about the same ratio with with lines of code written so um you know the situation wasn't getting better so so we said okay we have this Secrets problem we have a problem with introducing uh you know code vulnerabilities um and this is happening all over the platform and when you look at uh analyst data you know um they'll tell you kind of the same story in these big numbers and what jumped to me about this data is really this last bullet 81% of devs still choose to ship vulnerable code to meet deadlines and as security people I hear all the time talking with you um that you just feel like your devs aren't listening or they just don't care does anybody feel like that no sometimes few okay no one wants to admit it's all right I've I've heard people you know you get a few beers into people and they'll really Let It Go but it's okay you know sometimes uh your devs aren't listening to you right and and it's not that they're not listening to the advice you're giving them but they're inundated with so many things I mean a lot of stuff falls on a developer's plate all all day and as you know working at GitHub um we're we're advocates for developers we we really uh try to understand what's going on in the developers um day-to-day and how to help them and and we went out and talked to developers about why they're ignoring security vulnerabilities like what's going on that's causing them to just sort of like bypass you know any noisy alert you stick in their face um and they told us two really big things one is that there's two to many false positives um you know this industry went through this huge ground swell of VC investment and now there is an alert for every single thing that could possibly happen in code and developers are inundated with with thousands of these um you know if you work at a knock maybe you're like Ah that's my job I see lots of colors and I I like responding to that but developers don't and um they're you know they train themselves to be efficient and they see too much noise it it you know it's something they go and investigate and it turns out to not be a problem to them and then they just stop paying attention to it after that um the second thing they told us is that context switching is more expensive than people realize and I think as technologists we understand this but you know the frustration that developers go through when they're trying to remediate security vulnerabilities is sometimes really arduous like they don't have the right permissions to view the dashboard to understand what the alert is so they could spend a week trying to understand if a vulnerability that's being reported to them through you know maybe some internal tool that pings them on slack is really something that they have to care about and by the time they get to it it turns out to be a false positive or something that they don't care about and that whole cycle just discourages them from ever paying any attention to security alert so we've got to fix that right there's a user experience problem um that's going going on here so we uh we took kind of a multi-pronged approach here um there's an aspect of this where we need to introduce tooling and so we'll talk about some of the tools that we introduced um we also had to go and educate people and so part of uh me being here today and my wonderful marketing team uh giving me funds to come out here uh you know is helping do some of that so I'll help kind of share uh the goodness that we're working on so so that you're aware of it and and three we're building a security community so um I'll share some of the resources around how you can join that security community and what assets are available to you there um there's some really Nifty stuff we have uh on the platform I want to spend a little bit of time talking about three of the features that were on that um page and this is all free stuff on github.com so if you're an open source maintainer or you use GitHub uh you you this isn't like a sales pitch for you this is uh tools that are freely available um you know they're just buried in settings pages and so I want to come and kind of share where to find some of this um but the three big Investments we've made are in static analysis uh the first static analysis investment we've made is code scanning so this is uh an acquisition we made of a tool called seml uh seml is a static analysis security testing tool um it identifies you know insecure coding patterns in applications and it's built directly into GitHub so on every poll request we can run a static analysis check to say hey does this new code contribution introduce any vulnerabilities and if it does then we can take some action on it the second piece of our static analysis story and you know static analysis obviously because we're the SC we have all the codes sitting there at rest so we can look at it um so the second piece is uh secret scanning and and this was going to that first problem we talked about of having you know 300 plus tokens being leaked on the platform a day um we teamed up with AWS and sort of just a very informal way to say hey we're sure you're having this problem too how do we try to solve this let's run uh a regular expression you know tool like let's identify patterns and code uh as they're coming into GitHub and if we find them if we find API keys that match our our Rex pattern for them uh we'll we'll we'll tell you about it right and we'll start um notifying the users that hey their their key may be compromised and you know it's not just in the uh the the push that you're making to GitHub a lot of times this happens in the get commit hist so users are just unaware that the get commit history comes with the commit and so they'll think oh I tested you know this local database connection and everything worked great and then I deleted the key and then I published it um and now you you have a secret online so um we we worked with AWS to solve that so I'll talk about both of those um either I'll say code ql for code scanning or um secret scanning you know but these are kind of the big bucket features um the last piece that I'll talk about a lot is this product called dependabot again also free um dependabot is a tool that looks at all of the dependencies that are in an application and it compares the versions of those dependencies and transitive dependencies with known vulnerabilities uh for those packages and if we find that you're using a package that has a known vulnerability in it then we'll go ahead and alert you that hey you know you should upgrade this version or there's a patched version over here or um you know this has a cve that you should pay attention to so this kind of comprises the the big rocks in the story the rest of the stuff on that side was U more configuration settings so we'll talk about both of them but just kind of wanted to lay a baseline down of what what the tools are and what we're what we're talking about here as important as the tools are the way that the tools are built is also really important and so um I mentioned we talked to developers and we talked to a lot of developers about this security problem uh to kind of understand what we needed to do and so our approach to introducing all of these static analysis tools um was was really this kind of three three pillars here one is um developers want collaboration they need ways of asking questions without having to change screens and going to other places and so using some of the um you know the collaboration features built into GitHub um now if you turn on the security features it's very easy to escalate a poll request that's being blocked by a static analysis tool to a security team to say hey I don't understand what this alert is do I need to pay attention can you help me with this can you help me remediate it right so give them ways of of working in the community and you know contacting senior devs or or security leads on projects um the second big piece is they want it all in the platform right they don't want to go somewhere else so whether they're you know working in the web UI or the IDE you know as much as possible they want alerts delivered to them locally without having to navigate away to some other other location um and last they don't want to think about it they just kind of want it to be automated it should just be a thing in the background that developers don't have to care about and so um that was sort of our approach when we built all of these tools and I've had this graph this is one of our oldest you know views or oldest slides I've had when when having these talks with with customers um you know but but it tells the story just so clearly when you um when you automate and inject a security process into the developer's workflow they respond to it and that that's roughly what this graph is saying when you turn on dependabot and it automatically raises a poll request for you to update a version it's really easy for you to go yeah I do want to update that version I don't want to be exposed to that vulnerab anymore that's simple for people and they get it and they respond to it and you just see that engagement happen as soon as we start putting these tools in front of the developers in this way oops the second thing not the second thing sorry we we see this um with all of our tools uh so I talked about those three big tools and they've all basically experienced the same response rate from developers um when we went back and looked at static analysis scans of code um we saw that alerts were getting remediated you know maybe 15% of the time within the poll request right so if a tool blocked you or told you like hey you know this lint is off you should fix it U you know 80 plus perc of the time developers were just ignoring it when we started adding PR blocking uh scans in you developers respond to that right they they will remediate them 70 plus per of time you know in seven days which is really wild right um as security people this lowers the amount of alerts that we have to look at that a developer is claiming are false positives or that are you know being pushed into code and bypassing our security processes for us to go back and remediate later um we've also blocked a ton of these secrets and I'll talk about how we've done this in a little bit but um we we've gotten the the scans for API credentials down um in many cases to less than 1% false positive rates and so we've started blocking commits before they're written into the project so that keys don't get distributed to other copies of the the project when it's downloaded and so we've we've blocked roughly 20,000 of them um which is really exciting so um all of these tools you know are implemented in the same kind of way where we create uh you know automated poll requests wherever we can we we sort of run automated you know in in PR and provide as much context to developers you know natively in the platform and when we do that we get really good results out of it we started taking these tools that we were building and applying them to the open- source software and I think this is probably the first of what will be many QR codes so if you feel like taking pictures of the slides or you want um relevant blog posts that I'll link you to here feel free to pull those out and zap a picture um or or or follow the QR code but they'll be links to everything this one here goes to um the uh the security lab landing page that you can take a look at but we we you know through these Acquisitions we hired all of these Security Professionals and so we we put them in an organization called the security lab um and what they do is they use our static analysis tools to go out and look at open source projects and evaluate whether or not we find novel vulnerabilities in them um you know we do this you know 200 plus times a year but what's really exciting is that when you add the tools aspect that we talked about plus a little bit of helping hand from somebody that understands what's going on we can get that remediation rate up to 95% so you know 95% of the PRS that we make on projects to alert them about an open source vulnerability get fixed which is really really exciting so um this has allowed us to build uh the largest or maybe second largest database of uh open source vulnerabilities in the world uh so all of the vulnerabilities that we find we make freely available um on github.com advisories and this is just like nvd or other vulnerability databases you can pull it down freely um it's also what our dependabot tool uses to check whether or not any dependencies in your project have a vulnerability associated with it but that's been built up and curated through all of these research efforts um put on by the security lab okay I know I've been talking a lot give me a second here I get some water any questions [Music] yeah we report the CVSs number associated with it um so that that will come out um where we're headed with this longer term is trying to tell you whether or not uh that vulnerability is reachable um we can talk maybe more about that offline but that's sort of where the industry is going is like hey it's not only that I want to know that it's a critical but I want to know if it's a critical to you know and so we're working to solve that problem I can I can give you a little update there okay um so that was a lot about the like GitHub why you should care and and what we're doing to try to better secure the open source software that all of our daily lives depend on um but there's a lot that you can do as individuals to kind of help us on this journey and so the rest of this presentation is really kind of a Persona based approach to how do I use some of these tools um to to to improve Security on on GitHub or even in your in your own company um and the first thing I've got for you is please please please turn on two Factor authentication this is the single best thing that we can do uh to uh prevent people from taking over our accounts and to stop supply chain attacks who has tofa enabled on their GitHub account okay security room all hands let me tell you look uh if you glump GitHub in with like social platforms right you think about social platforms broadly and I'm you know this is some Twitter math that I did a while ago but uh what I he tell is that most social platforms have an adoption of around 2% toofa right that's really spooky um and and this is you know we're we're a bit better we maybe five seven 8X that right I don't know where we are now so don't take that as a hard number but um we're we're drastically improving and we um we want to do even better uh so by the end of this year anybody that contributes to github.com will have to turn on two-factor authentication it's no no longer going to be an option we're just going to force people onto this path um so that's a really exciting Milestone that we're moving to um this has taken three